Yesterday afternoon at approximately 4:30 p.m. we deployed a .htaccess redirect rule to support a newly-launched campus website. At approximately 4:45 p.m. our monitoring tools started informing us that some campus websites were going down and/or were redirecting to the newly launched website.
As we investigated, the scope of the issue quickly grew to include all campus SiteNow websites.
At this point, several things happened in quick succession: we issued an IT Service alert, identified the cause of the issue (improperly scoped redirect rule), removed the redirect rule, and pushed the change out for deployment across our infrastructure. While some SiteNow websites started recovering immediately, the fix took about 30 additional minutes to fully deploy across all SiteNow websites.
Though sites recovered at different times, our initial assessment suggests that the last site to recover from this issue did so after approximately 60 minutes from the start of the incident.
If you are still observing unexpected behaviors
If you continue to observe unexpected behaviors with your campus website this morning, your browser may have cached a faulty redirect from yesterday and is continuing to use it. This only affects people who visited an affected website yesterday during the outage and are attempting to visit it again before their web browser naturally refreshes its cached redirects.
To resolve this issue, you can follow the instructions for clearing your browser cache and cookies provided here: https://its.uiowa.edu/services/how-clear-cache-and-cookies-your-web-browser.
What happens next
The web team will continue to monitor campus websites to ensure there are no persistent issues. We will also conduct a thorough root cause analysis to ensure we understand why this happened and what we can do to prevent it from happening in the future. We will share our findings and updates with the web community.
On behalf of the web team, I apologize for yesterday’s outage. While we endeavor to provide exemplary service that provides over 99.9% uptime throughout the year, outages do happen and when they do, we invest in understanding the cause and learning how to avoid similar incidents in the future.
Please reach out directly if you have any questions or concerns.