Anatomy of Failure

So the title sounds a little foreboding – but we actually had a very successful run at the Maker Faire this past weekend! Over 50 people got to ride in the Dropship during the two-day event.

We had a situation where paying customers (one of who was autistic) were trapped for a couple minutes in the simulator, in the dark, with no AC, and no way to communicate, and no way to open the door. They handled the situation fine and gladly accepted a full refund but troubleshooting and responding to a potential emergency at the end of a long weekend is not something I ever want to deal with again.

Sequence of events:

  • 2015-06-27 18:24:24.732 – Simulator secured for the night, GNC on generator power
  • 2015-06-27 20:07:54.035 – Generator was manually switched off (unknown, unauthorized individual).  Event (“shore”) power was unavailable so GNC switched to internal battery
  • 2015-06-27 21:35:11.666 – Last entry before GNC was forced to shut down due to battery depletion (90 minutes later, as designed)
  • 2015-06-28 09:58:56.476 – GNC was force-ably* brought back online, internal battery now trickle-charging due to GNC operating load
  • 2015-06-28 17:16:28.796 – Last ride of the day started almost 20 minutes past the published operating hours
  • 2015-06-28 17:19:10.630 – Shore power was shut off, GNC switched to internal battery with only 4 minutes run-time estimated
  • 2015-06-28 17:19:28.063 – Last entry before GNC was forced to shut down due to battery depletion

Recommendations:

  • Advise Maker Faire of the security incident at 2015-06-27 20:07:54.035 (complete)
  • Install some kind of shield or screen on the front of the generator to prevent unauthorized access (PENDING)
  • *Do not override the UPS “minimum charge” to restart (when I arrived ~9:15 AM there was very little condensation on the ground from the AC indicating that the power hadn’t been on for very long… I became impatient for GNC to boot so I actually forced the UPS to come back online). There was a discussion that a delayed start to the day was a possibility and actually wouldn’t have impacted our revenue since the event was slow to start on Sunday. The EEPROM setting for this is 15% battery charge by default.
  • Indicate the low remaining run-time in amber (15 minutes) and red (5 minutes) on the electrical synoptic page (complete)
  • Environmental CAS message regarding low remaining run-time (less than 15 minutes since the current scenario is never more than 10 minutes) (complete)
  • With 5 minutes or less run-time remaining and on battery backup we should initiate the shutdown (complete)
  • Advise Maker Faire that they disconnected power with paying customers still in the box (previous day they came around and made sure that it wouldn’t be a problem) (complete)
  • Do not operate the attraction after the published hours OR switch to generator power before operating outside of published hours

One thought on “Anatomy of Failure

  1. Pingback: Maker Faire 2016 | Unistellar Industries, LLC

Leave a Reply