It's Friday, so the time has come for the weekly update post. Here's the latest:

What Happened This Week?

[1] AWS Migration (part 2)

As stated in the last update, the process of migrating the forums and blog to AWS is ongoing, and the estimate for wrapping that up at this time is Monday. Like I said before, migration is always a big thing that requires a lot prep and care, so it's slightly more time consuming than other things.

[2] Back to Work (Fixing)

Now that resources on the back end are beginning to be freed up, work to investigate and find solutions for items that are currently broken or otherwise not working properly has resumed. Priorities in this area include:

• Email Deliverability: Obviously this one is a biggie, and work is already underway to get this resolved. It's sadly something of a "chunky" issue, and involves multiple processes on the back end that have to be synced and made to work nicely - which can be challenging at times. Put simply, a bit of Overhaulin' may be needed (paging Chip Foose)
• Tags: Plenty of support tickets have come in regarding tag searching returning incorrect or no results, and so this has quickly climbed its way toward the top of the "to be fixed" list.  Work on this problem has begun, and we'll hopefully have some good news soon.

[3] Back to Work (Adding)

Work is also beginning to start back up on updates and new things for the website. A small example in this area is the email widget that has been added to the forums on the upper left hand side of the page. As stated last week, the roadmap is already situated, it just comes down to the actual "doing". Look for more in this area as we go forward.

~~

That's going to about do it for now, more to come, stay tuned.

It's Friday, which means that as promised, it's time for the weekly update post on what has been going on behind the scenes. This particular post will cover roughly the last month rather than the last week since I have scrapped my previous plan of a larger blog post concept and reverted back to the better-for-the-community weekly forum topics. So, without further delay, let's get into it.

What's Been Going On?

[1] Site Stability Stuff

The work on fixes, features and bugs that had been taking place (as seen in the previous weekly updates) had to be put on hold temporarily due to the re-emergence of malware and bot activity on the forums. This required various updates and fixes to implement bot mitigation that would prevent these malicious activities and keep the site safe from the wild trolls of the web. While this doesn't sound like a lot initially, it's a bit more of an involved process, since it takes examining the modes of infiltration and attack, doing appropriate research, testing fixes, and finally rolling them out.

This process is finally wrapping up and is due to be completed, which will allow other work to resume as it had been prior to the interruption.

[2] AWS Migration

The task of migrating the forums and blog over to AWS has begun (and should be wrapped up by end of next week). This move will put all of Obsidian Portal in "one" place, and should also add additional security to the stability fixes that have been enacted. Migration is always a serious matter that must be done carefully, and it takes a significant amount of lead up and prep work to ensure that it goes smoothly. The upsides are good: the flexibility, scalability and reliability of AWS are well worth the effort, and will increase site stability.

[3] Sign offs and Roadmap

While actual work on fixes, features and bugs had to be stalled, the planning and ironing out of the roadmap for the upcoming development cycle was not impacted. The plan is now together and the appropriate sign offs for upcoming bug fixes and the like are completed. Examples on bugs and problems to be addressed include, but are not necessarily limited to: email deliverability, interactive maps, tag searching and functionality on mobile.

[4] Getting the new community manager up and rolling

Like the Death Star, there's been some work involved in getting me "fully armed and operational" as it were. Each week has brought new challenges and responsibilities, but it's finally coming together. Look out Alderaan!

[5] Front Facing / Nuts and Bolts

Some of the more routine or mundane tasks that were behind or not being kept up on have been reassigned, and as such these are now being completed once again. Examples include:

• Campaign of the Month Process - The Campaigns page has been brought up to date with the missing CotM winners, and all associated badges have been awarded. Obviously we're still behind on CotY and there hasn't been a CotM since December, but this too shall be addressed. Vacant committee seats are being filled, and CotM will be ramping up once more in the reasonably near future.
• Game systems - Requested game systems are finally back to being added to the site. Recent additions include: Genesys, Firefly, Starfinder, and Coriolis: The Third Horizon. This is something that has been missing for a while now, and it's great to have it back.
• DST review/approval - Another item that has been addressed and is now back in action, we should all begin reaping the benefits of new sheets as time goes on.

Well, that'll about do it for this post. I tried to keep it reasonable to read while covering everything to at least some degree. If you have any additional questions or concerns, post here. Outside of that, more updates to follow next Friday.

We have some exciting news!

We now have an official community and content manager here on Obsidian Portal – Dan (aka @Kallak) As a long-standing member, a member of the inner council and being responsible for releasing the Campaigns of the Month on the blog, he was a natural choice.

I would like to officially welcome @Kallak as the shiny new Obsidian Portal community manager - you’ll be seeing more content from him here on the forums as well as the blog in the near future.

Plus he’ll keep you posted on the new features and tech updates we’re working on to take Obsidian Portal to the next level. If you’ve got content ideas for the site – please share them with @Kallak!

Thanks guys!

Jameela

Just wanted to drop a quick note that we made some changes to the order of the forum threads so that the latest posts show up first and work backwards from there.

We just released the changes this morning, let us know what you think of this new ordering system.

Thanks!

Jameela

Thanks for your patience and I'm glad to see that things have been running smoothly for everyone the last few weeks.

I'm dropping an update to let everyone know that we have been working on staging to get the issue with the avatars and the media subfolders resolved. These have been corrected on staging and we are knocking out the final tweaks to our deployment process to make sure the release smoothly.

We expect to be able to push these changes live before the week it out.

Thanks for your continued support and patience. 

If you have any other technical issues that have not been addressed please submit a support ticket to our help desk.

Thanks!

Jameela

Fixes 1 and 2 that were done and outlined in the Overview of fixes from the last 60 days, should have resolved the problem with the tags and search functionality, the moment the fixes was applied we were monitoring this function and it appeared to work for a while.

When tested again yesterday, we did get an error, though it appeared to still be working on the OPAdmin account, so we have deduced that it seems to be affecting users at random (yay for the bundle of wires getting more knotted), we are still investigating the cause behind this sporadic behavior.

We're hopeful that if we can get someone in for a few quick server fixes that this should resolve the intermittent activity and these two functions. We'll keep everyone updated as more details come to light.  

Thanks!

Jameela

I know that there hasn't been much activity in regards to communication on the forefront, I want everyone to know that this didn't mean we weren't paying attention. We wanted to share with everyone an overview of the challenges we ran into over the last 60 days and what we have done to help fix certain features.

1. An unexpected issue with async server, which has been fixed, but this had been causing an issue with background jobs such as billing, search, tagging, achievements and maps processing. This is the reason tags were intermittent or not working for some, and why maps took so long to process.

For those of you that appreciate more technical details, the problem itself was specific to one of gems called and the MySQL database. Due to the amount of delayed jobs, the gem was not able to process them all, and the MySQL server ended up blocking entries, we applied a code fix and updated the gem configuration to resolve this.

2. In addition to the functions above, the server also stores data related to wiki's and campaigns, so we increased storage space on the server so it has capacity for growth, and adjusted the search engine settings to handle the changes. We are periodically monitoring it to make sure there is enough space so that it continues to run.

3. Another instance of the server going down, which occurred twice last month, was related to a violation of our servers terms of use, related to content covered under copyright laws being hosted on the site. In addition to resolving this we upgraded to a new server size and generation with better security hardware.

4. In the last week we had intermittent services, this was due to a higher amount of traffic, we again increase our server size with higher amount of CPU capacity, plus adjusted the server configuration to handle the growing traffic better.

I hope these details help to shed some light on what the last 60 days has been like, if anyone has any feedback, suggestions on how we can continue to improve the technical infrastructure we are open to listening.

Thanks!

Jameela

As you can see we started a new category :)

The purpose of this category, as the name suggests, is to supply the community with a weekly update on what we’re doing to improve Obsidian Portal.

I’m going to start with acknowledging the concerns that have been expressed in the various threads.

1. The website being intermittent and having gone down earlier in the month.

The server had been taken down early last month on account of an oversight on our part, related to content that was covered under copyright laws. We did receive a notice and we did remove most of the content, but apparently we didn’t get it all and the server was taken down because of it. Please be mindful of the content you use in your campaigns moving forward so we don’t have these issues again.

Regarding the intermittent activity that occurred the last couple weeks, we had an expected increase in traffic load, which the server was not able to handle, we have since (as of March 1st) upgraded the server, which should resolve the errors that were occurring on updates.

 

2. Steps we are taking to address the core functions in need of repair.

I’m not going to nor do we need to list them out again, we will use this thread as a reference

https://forums.obsidianportal.com/discussion/4894/obsidian-portal-changes-and-updates-for-2018/p1

I have asked our developer to give me an in-depth overview of what needs to happen for each of these elements to be fixed so they can function as they once did. Once I have that from him I will share a fresh update with those details.

3. The Help Desk

We appreciate all of you that have been submitting tickets and we sincerely apologize for the lack of response. We have found some technical issues that created problems with how the tickets were routed, so they were not being filtered and seen properly. This has been fixed and we will be addressing/catching up on tickets over the coming weeks.

4. Staff Changes

As I’m sure many of you noticed, we have had many come and go over the years. We did have a dedicated team working on the site a year or so ago, but have since let them go as they did not perform at the level we expected or needed. Additionally, there is what we consider to be, mistreatment of the community, and there are a number of you that refer to this as Obsidian Fallout where users were banned and posts pertaining to tech issues removed. While we do reserve the right to remove posts, it should not have been done in such a manner that created a rift between us and long standing members. I can do nothing more than apologize for the behavior of these ex-team members and assure you once again that we will be transparent with everyone on updates, changes and issues moving forward.

 

5. We are Hiring!

We are actively looking for a couple of worthy candidates that can help us manage and support this community, with the opportunity to possibly branch into some of our other business units as well. I will be sharing more information regarding what the job would entail in the next 24 - 48 hours as well as instructions on how you can apply.

Finally, now that we have our support system functioning properly, I would like to ask everyone to please send any technical requests or account issues there. If you have tickets that are already submitted, please reply to those in lieu of creating a new one so that we don’t get bogged down with duplicates of the same requests.

Please continue to share your thoughts and ideas with us, more to come tomorrow!

Thanks everyone!

Jameela

Navigation that doesn’t give 502 errors every other week. 

You have the word portal in your name and all we’ve gotten for our money this year is bad gateways. Or at least change your name to 502Portal or something. I feel like that would bring this a little closer to being the ultimate destination for tabletop gamers. 

As I’m sure you all know we are in need of someone who can help with community management, especially now that our technical team is finally in place. There have been several of you that reached out to us offering to help and we definitely appreciate that. We want to make an official request to those of you here that would be interested in helping us with Customer/Community Support, this would entail tasks such as but not limited to:

• Moderating the forum and replying to posts when appropriate/directing posts to the help desk when needed, keeping it on an even keel, without any unnecessary drama. Encouraging people to ask questions, share their knowledge, engage new subscribers etc. 
• Replying to support requests that come through the Zoho Helpdesk that are related to how to use OP, ensuring any tech questions go to the tech team side to handle.
• Monitoring and replying to posts and questions on the social media channels (Facebook & Twitter), posting new and relevant content via these social channels, and looking for opportunities to encourage content creators to come forward.
• Serving as a conduit between tech team, community and support to help us fill the gaps and get ahead and stay ahead of any general issues and look to add features, more support etc. Generally be a positive contributor and driver of what we need to do and how we can improve the site for everyone.
• Help create a new FAQ center and develop the forum in more interesting ways.

We would like to start someone from the community who plays, preferably an active Ascendant Member, as being on the platform gives you insight to helping the people coming to it and to help us take it to the next level.

Shoot us an email with the subject "OP Job Opportunity" and we’ll connect with you for more info.

Thank you!

Jameela

Is there anything i can do?

Just wanted to drop a quick update on progress so far, we have a couple guys testing the server on staging and are tweaking it to stop the fails. We suspect that the issue has to do with the environment as we did have to increase our server sizes recently to eliminate the time outs, we did adjust the configurations on the system that handles campaign, character creation etc which temporarily eliminated the errors, but as everyone has noted they are still persisting. 

We pushed a change live today after stress testing it to see where it failed, unfortunately once live it continued to produce errors so we pulled the server that we thought would have been fixed (worked on staging so we were optimistic it would do well in production), we then pushed it back on to gather additional data and to see if we could identify any trends to help us pinpoint the issue. 

Our team is still trying a few things in the dev environment to eliminate these errors, though at this time I don't have any positive news, I can assure you that we're working diligently to get this resolved.

I should have more info to share tomorrow and will keep everyone posted here.

Thanks for your patience!

Jameela

We have some good news, as of yesterday evening we signed on a top notch development team that will be working with us to help resolve the immediate issues that are currently plaguing campaign and character creation (the dreaded 500 error that keeps appearing), we suspect that our efforts here should also resolve the issues with tagging and search as well as the avatar uploads.

They just started tackling this today, so there was time needed to set up a staging environment, getting the credentials in place to access the staging server and all that other set up work. We have already been able to recreate the issue so now we're in the process of testing and fine tuning the server so that these issues no longer occur.

We are only working in staging at the moment but have made some great progress in identifying the issue, now we're in the process of trying different patches, once we have a solid fix in place we'll push everything live. 

Thanks so much for your patience in the meantime and we'll drop an update once we have more progress to share.

Constructive feedback and ideas are welcome.

Thanks again!

Jameela