iBill Update

[MarkIBill]

Due to Hurricane Francis, we have just recently restored power from our Power Utility at our corporate offices. We lost power on Saturday and switched to our diesel generator. Unfortunately, Sunday morning we lost our generator and had to wait to get a spare part. This was one of the largest Hurricanes to ever hit the states so there were many people and businesses affected. At that time we switched to a secondary data center site in Atlanta. We processed signup transactions at our secondary site until about 1:30 am this morning when we were able to restore our generator. CMI and iBillcs were not available during this time as we were running only essential operations (transaction processing). We did this to conserve fuel as the status of available fuel was questionable due to the number of people and businesses affected by the storm. More than 6 million people were without electricity. Our operations and IT staff has been working around the clock for many days now. We have been working to restore all ancillary services. CMI and iBillCS are now online.

During this time, consumers were given a splash page explaining that our office was temporarily closed and some web services were affected. They were asked to send an email to [email protected]. We will be working diligently to respond to all of those requests including issuing refunds for people that were not able to immediately cancel.

We appreciate our client?s loyalty and patience during this difficult time.

Mark

[lorsha]

Mark honestly......

How a company of your size, could NOT have another plan for this situation; is absolutely unacceptable.

I'm sorry....but that's how I feel....

[Dawgy]

our ibill sales were normal. but zero affiliate sales, as the affiliate links did not work. we were also able to get into ibillcs, etc... people clicking affiliate links for our sites got a rather vague hahahahahahahahahaha looking error page.

last i heard it was 2.5 million people without power

[Steen2]

Originally posted by lorsha
Mark honestly......

How a company of your size, could NOT have another plan for this situation; is absolutely unacceptable.

I'm sorry....but that's how I feel....

[Nysus]

Originally posted by lorsha
Mark honestly......

How a company of your size, could NOT have another plan for this situation; is absolutely unacceptable.

I'm sorry....but that's how I feel....

IBill processing was fully down for awhile?

Matt

[scardog]

Mark - Can you bring us up to speed on Yesterday's signups and rebills database ? When do you think that will be available?

[MarkIBill]

Originally posted by scardog
Mark - Can you bring us up to speed on Yesterday's signups and rebills database ? When do you think that will be available?

You should see both tomorrow sometime.

[rowan]

Affiliate links didn't work? Ouch! Why wasn't that considered part of the backup transaction system?

[XPays]

hi mark-

glad to see all the joins flowing as normal and it was very akamai-ish to have the join pages up even if local power was down

i suggest that all web companies consider natural disasters in their hosting strategies. obviously using separate hosting for join pages was a good plan to keep initials coming in.

we have several vendors hosting in florida that we have had issues with due to the last 2 hurricanes. hosting in florida is an annual invite for downtime imho. that being said we have servers in earthquake ridden areas

it is tough to balance natural disaster prep for internet commerce. anyway, thanks for keeping our joins live.

[MarkIBill]

Originally posted by Dawgy
our ibill sales were normal. but zero affiliate sales, as the affiliate links did not work. we were also able to get into ibillcs, etc... people clicking affiliate links for our sites got a rather vague hahahahahahahahahaha looking error page.

last i heard it was 2.5 million people without power

I will check into this for you. Please send me your account number. You can send it to [email protected]

[Major (Tom)]

Originally posted by lorsha
Mark honestly......

How a company of your size, could NOT have another plan for this situation; is absolutely unacceptable.

I'm sorry....but that's how I feel....

I agree. I'm not going on an anti I Bill rant here, but... having a hurricane take down a company is something that happens to a mom and pop operation. What if there was an earth quake, or a terrorist attack. I mean you guys need to plan for this stuff. It's not like hurricanes are a rare event in florida. That's like building a house near a volcano in hawaii, and wondering why it caught fire when the lava started flowing. Should'nt there be a system of redundancy in place? Millions of dollars go through I Bill a month, it's a shame that some wind makes everything go tits up. Well, with that being said, it's obvious what needs to be improved over there.
Ditch those web good pages too while you're at it lol.
Duke

[XPays]

Originally posted by rowan
Affiliate links didn't work? Ouch! Why wasn't that considered part of the backup transaction system?

fyi- obviously that did not affect XPays as we use our own patented software.

[MarkIBill]

Originally posted by rowan
Affiliate links didn't work? Ouch! Why wasn't that considered part of the backup transaction system?

Yes, it was part of our essential processing systems. If you had an issue please forward it to me.

[directfiesta]

It was a big m..th..r fuc...r storm.

Give these guys a break.... I switch my loin to my secomnd processor in Europe. Once eveything is fully normal, I will change back to Ibill...

No reason to get upset ....

Ibill is way better than a year ago ....

[XPays]

if any fed's or isp's see the epiccash logo under the handle "XPays" please be advised:

GFY put that there and XPays is not affiliated with all of the epiccash and epicdvd illegal spam and spoofing of hotmail, msn, yahoo etc...

[MarkIBill]

Originally posted by DukeSkywalker
I agree. I'm not going on an anti I Bill rant here, but... having a hurricane take down a company is something that happens to a mom and pop operation. What if there was an earth quake, or a terrorist attack. I mean you guys need to plan for this stuff. It's not like hurricanes are a rare event in florida. That's like building a house near a volcano in hawaii, and wondering why it caught fire when the lava started flowing. Should'nt there be a system of redundancy in place? Millions of dollars go through I Bill a month, it's a shame that some wind makes everything go tits up. Well, with that being said, it's obvious what needs to be improved over there.
Ditch those web good pages too while you're at it lol.
Duke

Since when did it take us down? We scaled back operations but took processing during this period. While we experienced issues there was always another plan. For example, we have a 500 gallon diesel generator that gets tested every week. Yet, it can break down and did after an extended period. Unfortunately, it can be little difficutl in the middle of a rather large storm to get it repaired. Thus, we cut over to a back up pre-planned and built out facility out of state. I can't give all of the details of what happened this weekend but there were a lot of good people working hard to keep things running during a very tough time.

[EscortBiz]

wishing you guys all the best!

[tony286]

Thanks ibill for keeping us posted and keeping those transactions flowing.

[Jake]

Originally posted by MarkIBill
.....CMI and iBillcs were not available during this time as we were running only essential operations (transaction processing).....
Mark

You mean "subscription" processing, not "transaction" processing, right? Our iBill store was down for over 48 hours and would not process transactions. Anytime a user clicked a "buy" button in our store they were taken to a page that stated:

"...Transaction processing is NORMAL and all transactional data is secure, as a result of our backup servers in an out of state location..."

Well, transaction processing quite obviously was NOT NORMAL if the user is taken to this message when they attempted to buy something.

If iBill does not consider tangible processing to be part of their "essential operations" then maybe they shouldn?t offer a billing solution for tangibles.

Needless to say, I'm more than a little disappointed at how this whole situation was handled.

[Shoplifter]

Good to hear it's working out. Do you have mains power back now?

[Nasty]

Well I cant complain, since ibill coming back on line, I received several ibill signups, bam, bam, bam....

Actuallly a record ibill day for me

on another note, I am running around 0 in 10,000 with nastydollars on se traffic for the past 24 hrs

[FaberX]

I'll wait until tomorrow to see how many signups show up, however:
- I asked 5 friends to try and signup from different states, NONE of them could. Error recieved was bank could not be reached
- Affiliate redirects DID NOT work, so most of my inbound traffic from them was lost, if not all.
- Because of this, as surfers didn't get to my tour, no consoles worked either, more revenue lost.

I'm expecting, and guess most people processing with ibill should demand compensation for this lose. It's not like they are a tiny company, so what happened is at least to me, unacceptable.
I'm in SoFla, and even though the storm was big, I can't imagine what could have happened if it was as big as it was expected 2 days earlier.

[BiggleJones]

Marc...I'm sorry, but this is bullshit. You guys make a fuckin killing on your %'s....buy some backup servers in other areas (that will not shut down due to Tornados, Earthquakes, quicksand...etc).

[quiet]

good to see the cmi back up...

[Giorgio_Xo]

Most large financial institutions headquarter their back office operations in Arizona since it is mathmatically the least likely place for a natural disaster in the United States.