Service	Status	Notes
Cooling
Electrical
Email (SMTP)
Network
VoIP
VPS
Web (HTTP/S)

VM Hosts					Dedicated Hosts		Infrastructure
pdxvmh01.tocici.com	pdxvmh02.tocici.com	pdxvmh03.tocici.com	pdxvmh04.tocici.com		pdxded01.tocici.com	pdxded02.tocici.com	pdxigw01.tocici.com
pdxvmh05.tocici.com	pdxvmh06.tocici.com	pdxvmh07.tocici.com	pdxvmh08.tocici.com		pdxded03.tocici.com	pdxded04.tocici.com	pdxigw02.tocici.com
pdxvmh09.tocici.com	pdxvmh10.tocici.com	pdxvmh11.tocici.com	pdxvmh12.tocici.com		pdxded05.tocici.com	pdxded06.tocici.com	pdxisw01.tocici.com
pdxvmh13.tocici.com	pdxvmh14.tocici.com	pdxvmh15.tocici.com	pdxvmh16.tocici.com		pdxded07.tocici.com	pdxded08.tocici.com	pdxisw02.tocici.com
pdxvmh17.tocici.com	pdxvmh18.tocici.com	pdxvmh19.tocici.com	pdxvmh20.tocici.com		pdxded09.tocici.com	pdxded10.tocici.com	pdxssw01.tocici.com
pdxvmh21.tocici.com	pdxvmh22.tocici.com	pdxvmh23.tocici.com	pdxvmh24.tocici.com		pdxded11.tocici.com	pdxded12.tocici.com	pdxssw02.tocici.com
pdxvmh25.tocici.com	pdxvmh26.tocici.com	pdxvmh27.tocici.com	pdxvmh28.tocici.com		pdxded13.tocici.com	pdxded14.tocici.com	pdxeye01.tocici.com
					pdxded15.tocici.com	pdxded16.tocici.com	pdxeye02.tocici.com
					pdxded17.tocici.com	pdxded18.tocici.com	pdxhtp01.tocici.com
							pdxhtp02.tocici.com

• All times are US/Pacific.
• The following symbols signify the most severe issue (if any) encountered during that day:

1. Our regular maintenance windows are between 9pm-11:59PM (US/Pacific) every evening. Exceptions are made for urgent/emergency issues.
2. Most maintenance has no impact on production services, if we expect any impact we'll post a notice here, and via an email broadcasts.

These four features are currently under development, at this time (20MAY2010) the whois function is fully-functional, the others are being adjusted for production usage.

• Visual Looking Glass

IP or Server FQDN:  

IP or Server FQDN:  

Domain:  

IP or Server FQDN:  

Following server are involved in the search:
blackholes.mail-abuse.org, dnsbl-1.uceprotect.net, dnsbl-2.uceprotect.net, dnsbl-3.uceprotect.net, dnsbl.njabl.org, dynablock.njabl.org, bhnc.njabl.org, dnsbl.sorbs.net, sbl.spamhaus.org, xbl.spamhaus.org, zen.spamhaus.org, sbl-xbl.spamhaus.org, smtp.dnsbl.sorbs.net, socks.dnsbl.sorbs.net, relays.mail-abuse.org, blackholes.mail-abuse.org, postmaster.rfc-ignorant.org, dsn.rfc-ignorant.org, abuse.rfc-ignorant.org, whois.rfc-ignorant.org, bogusmx.rfc-ignorant.org, cbl.anti-spam.org.cn, block.dnsbl.sorbs.net, no-more-funn.moensted.dk, cblplus.anti-spam.org.cn, dynamic.tqmrbl.com, relays.tqmrbl.com, clients.tqmrbl.com, bl.spamcop.net, l1.spews.dnsbl.sorbs.net, l2.spews.dnsbl.sorbs.net, combined.njabl.org, dnsbl.ahbl.org, blacklist.spambag.org, opm.blitzed.org, list.dsbl.org, multihop.dsbl.org, unconfirmed.dsbl.org

Day@Time	Service	Notes
31AUG2010@22:00	VPS	Incident-free kernel upgrades completed, on schedule!
29AUG2010@23:00	ALL	Incident-free kernel upgrades completed, on schedule…laying the groundwork for some exciting announcements soon!
08AUG2010@22:00	ALL	Incident-free electrical upgrades completed, on schedule; all eight BladeCenter power supplies are now online.
02AUG2010@00:01	VPS	Scheduled hardware upgrade in PDXVMH18 has been completed. 2nd IBM BladeCenter Chassis is now online.
22MAY2010@22:55	VPS	Scheduled hardware upgrades in VPS host servers have been completed.
14MAY2010@21:35	VPS	Replaced problematic hardware, without service interruptions.
25MAR2010@20:00	VPS	VPS host server replacement completed (failed baseboard management controller).
19FEB2010@09:00	VPS	Completed replacement of PDXVMH10 (failed power supply).
07JAN2010@09:00	Electrical	The city of Portland experienced a major substation fire and subsequent 8 hour long power outage throughout most of the downtown area on January 7th 2010. Included in the outage area was our datacenter. Although thanks to the redundant power provided via the backup battery bank and two large diesel generators, TOCICI had zero service interruptions.
28DEC2009@00:05	ALL	Whew…time to go home; just completed the installation of a few new servers, another network switch, and a backup IP-KVM. YAY for scheduled maintenance windows!
09DEC2009@21:00	VPS	The kernel panic appears unrelated to any hardware issues. We'll continue to monitor the situation, but we're convinced that the kernel panic was a non-reproducible software fluke.
09DEC2009@02:55	VPS	A kernel panic on PDXVPS10 forced a watchdog restart on . Contingency plans in effect; while the machine is back online, and all VPSes are running, we've taken the primary network interface offline, the secondary is now servicing network traffic. In addition, we're running diagnostics on the primary network port to determine if its a hardware or software quirk which caused the kernel panic.
07DEC2009@21:00-23:00	Network	Scheduled network switch replacement completed without complications.
29NOV2009@10:27	ALL	All systems consolidated and online! Thanks for the awesome data circuit Stephouse!
29NOV2009@09:00-11:00	ALL	This morning we'll be finalizing the last phases of our system consolidation project
28SEPT2009@03:14	VPS	We just completed the migration of three existing Google Health connected apps to our infrastructure. WOOT!
10SEPT2009@16:49	Network	Latency issues seem resolved - we're staying on top of this folks!
05SEPT2009@13:25	Network	Latency continues to settle back into historical norms, but we're still not fully-satisfied
03SEPT2009@11:19	Network	Our historical graphs are showing a very undesirable trend of increasing latency - looks like its time for a new bandwidth provider
10AUG2009@19:00-23:59	Network	Primary network provider is replacing their Portland/Salem long-haul cable; expect intermittent 1-5minute network interruptions throughout this evening.
08AUG2009@07:03	Network	After a brief unscheduled provider/network outage, we're back online w/ latency back to pre-August 7th trends. WOOT!
07AUG2009@14:17	Network	Latency via two back-hauls remains unusually high
07AUG2009@12:19	Network	Latency is creeping up again - nothing critical, but we're not amused. Broadcast email details forthcoming.
02JUN2009@07:23	Network	latency graphs now trending back to normally low levels
01JUN2009@09:37	Network	Crazy days ahead…latency graphs are showing an unexpected upward trend
07JAN2009@16:24	ALL	Our equipment is being migrated to the Pittock; an impressive datacenter in downtown Portland with three redundant commercial power feeds, diesel beasts providing backup power, and a very intimidating battery bank. Watch our upcoming broadcast email for more info.
18DEC2008@03:00-06:00	VoIP	VoIP equipment upgrades are going in…watch email for details.
03NOV2008@23:12	Cooling	82F in the datacenter…we're raising hell, and looking for alternative spaces
15OCT2008@08:27	Cooling	75F in the datacenter. Unless this is resolved soon, we're packing up our equipment and moving out.
02OCT2008@14:48	Cooling	77F in the datacenter. Still leaning on the provider to do something about their cooling failures.
17SEPT2008@16:47	VPS	…aaaaannnnddd POP! goes the disk drive. Thank goodness for redundant hotswap equipment; emergency equipment replacement in PDXVPH07 completed w/ zero downtime
17SEPT2008@16:32	Cooling	79F in the datacenter…aaaaannnnddd, yep, the disk drive is dead…
04APR2008@07:16	ALL	This morning we were rudely awakened to screaming phones & pagers…our colocation provider was caught in a floor-wide power failure, not once, but twice (6:27am & 6:52am). So much for promises of redundant power! We're onsite now, and it's amazing how many iPhones are around…scary…the room is full of very upset SysAdmins. Yet-again, the datacenter space is far warmer than acceptable for a datacenter. Details forthcoming via an email broadcast.
27MAR2008@21:12	Cooling	Our temperature probes are quite upset…although not nearly as upset as we are; it's nearly 84F in the datacenter
13MAR2008@13:01	Network	reverse DNS (PTR) has finally been restored. Service provider has given us a generous refund which we're now passing onto the affected customers. Watch your email for respective partial service refunds.
12MAR2008@08:12	Network	reverse DNS (PTR) is still not functioning as promised via contracts…we're very unhappy with the service provider, and making that well known. Also seeking alternative providers now.
11MAR2008@08:14	Network	Our network provider has FUBARed reverse DNS (PTR) records - working with them to resolve
19FEB2008@12:05	VPS	Just added 5TB to our storage array - yay for growth!
03AUG2007@12:37	VPS	We're adding 1TB of additional storage capacity to our infrastructure
22JUL2007@23:53	Network	Whew! Systems migrated completed just seven minutes before the deadline. Enjoy the new services folks!
20JUL2007@08:15	Network	It's time to say goodbye to a very unprofessional and frustrating hosting provider. We're growing, we demand service excellence, and we will not tolerate the sillyness of needing to call the CEO's personal cell phone for resolution of frequent and fundamental core competency failures. Details forthcoming in emails to affected customers.
06FEB2006@11:19	Network	ICMP RTT latency through two of our three multi-homed routes has skyrocketed…network provider is working to resolve
30JUL2005@14:18	ALL	Looks like our colocation provider forgot how to use copy/paste … FUBARed reverse DNS is being resolved now
29JUL2005@23:47	ALL	Systems migrations completed w/out issue.
28JUL2005@14:18	ALL	We're finally reaching “non-low-budget” status - equipment migrations will commence soon!