Last In - First Out

From the Minneapolis Star Tribune:

“A hacker attack at payroll processing firm Ceridian Corp. of Bloomington has potentially revealed the names, Social Security numbers, and, in some cases, the birth dates and bank accounts of 27,000 employees working at 1,900 companies nationwide”

A corporation gets hacked, ordinary citizens get screwed. It happens so often that it’s hardly news.

This is interesting to me because Ceridian is a local company and the local media picked up the story. That’s a good thing. I’m glad our local media is still able to hire professional journalists. The executives of a company that fail like that need to read about themselves in their local paper and watch themselves on the evening news. They might learn something. If we’re lucky, the hack might even get mentioned at the local country club and the exec’s might get a second glance from the other suits.

We aren’t that lucky.

In a follow up story, the Star Tribune interviewed a man who claims that he has not had a relationship with Ceridian for 10 years, yet Ceridian notified him that his data was also stolen. The Star Tribune reports that Ceridian told the victim that the compromise of 10 year inactive customer data was due to a ‘computer glitch’:

“a Ceridian software glitch kept it in the company's database long after it should have been deleted.”

Sorry to disappoint the local media, but computer glitches are not the reason that 10 year old data is exposed to hackers.

Brain dead management is the cause.

But even brain dead management occasionally shows sings of life. According to the customer whose 10 year old data was breached:

"The woman from Ceridian said they're working on removing my information from the database now,”

Gee thanks. What’s that horse-barn-door saying again?

Given corporate America’s aversion to ‘DELETE FROM…WHERE…’ queries, my identity and financial information is presumably vulnerable to exposure by any company that I’ve had a relationship with at any time since computers were invented.

That’s comforting.

The future:

Cameras will be ubiquitous. Storage will be effectively infinite. CPU processing power will be effectively infinite. Cameras will detect a broad range of the electromagnetic spectrum. The combination of cameras everywhere and infinite storage will inevitably result in all persons being under surveillance all the time. When combined with infinite processor power and recognition software, it will be impossible for persons to move about society without being observed by their government.

All governments eventually are corrupted and when corrupted will misuse the surveillance data. There is no particular reason to think that this is political party or left/right specific. Although it currently is fashionable to think that the right is evil and the left is good, there is no reason to think this will be the case in the future. The only certainty is that party in power will misuse the data to attempt to control their ‘enemies’, whomever they might perceive them be at the time.

Surveillance advocates claim that cameras are simply an extension of law enforcements eyes and therefore are not a significant new impingement on personal freedom.

I disagree.

Here’s how I’d build a surveillance system that allows the use of technology to maximize law enforcement effectiveness, yet provide reasonable controls on the use of the surveillance against the population as a whole.

• The cameras are directly connected to a control room of some sort. The control room is monitored by sworn, trained law enforcement officers. The officers watch the monitors.

So far this is ordinary surveillance. Here’s how I’d protect individual privacy:

• The locations of the cameras are well known.
• All cameras record to volatile memory only. The capacity of the volatile memory is a small, on the order of one hour or so. Unless specific action is taken, all recorded data more than one hour old is automatically and irretrievably lost. A ring buffer of some sort.
• If a sworn officer sees a crime, the sworn officer may switch specific cameras to non-volatile storage. The action to switch a camera from volatile to non volatile storage is deliberate and only taken when an officer sees specific events that constitute probable cause that a crime is being committed, or when a crime has been reported to law enforcement. Each instance of the use of non-volatile storage is recorded, documented and discoverable by the general public using some well defined process.
• Once a camera is switched to non-volatile storage, it automatically reverts to volatile storage after a fixed time period (one hour, for example), unless the sworn officer repeatedly toggles the non-volatile switch on the camera.
• The non-volatile storage automatically expires after a fixed amount of time (24 hours, for example). If law enforcement believes that a crime has occurred and that the video will be evidence in the crime, law enforcement obtains a court order to retain the video evidence and move it to permanent storage. The court order must be for a specific crime and must name specific cameras and times.
• When a court so orders, the video is moved from non-volatile storage to whatever method law enforcement uses for retaining and managing evidence. If the court order is not obtained within the non-volatile expiration period, the video is irretrievably deleted. If the court order is obtained, the video becomes subject to whatever rules govern evidence in the legal jurisdiction of the cameras.

In the case of a 9/11 or 7/7 type of event, the officer would simply toggle all cameras non-volatile mode and would continue to re-enable non-volatile mode every hour for as long as necessary (days, if necessary). The action of toggling the cameras would be again be recorded, documented and discoverable.

To prevent the system from being subverted by corrupt law enforcement, (think J Edgar Hoover and massive illegal surveillance) the systems would be physically sealed, the software and storage for the non-volatile and volatile storage would be unavailable to law enforcement.

There would be some form of crypto/hash/signing  that enables tracking the recordings back to a specific camera and assures that the recordings have not been altered by law enforcement.

The key concepts are:

1. the system defaults to automatically destroying all recordings automatically.
2. a sworn officer of the law must observe an event before triggering non-volatile storage.
3. specific actions are required to store the recordings
4. those actions are  logged, documented and discoverable.
5. a court action of some sort is required for storage of any recording beyond a short period of time.
6. the system would be tamper-proof. The act of law enforcement tampering with the systems to defeat the privacy controls would be a felony.
7. the system would maintain the integrity of the recordings for as long as the video exists.

And most importantly, the software would be open source.

---

EPIC Video Surveillance and Wikipedia contain quite a few other thoughts on surveillance.

I don't have an opinion (yet) on Google's new DNS service, but I have to admit that they snagged some pretty cool IP addresses for their public resolvers:

8.8.8.8
8.8.4.4

DNS is one of the few services that benefits from a memorizable IP address.

Google chose wisely.

“imitate the superficial exterior of a process or system without having any understanding of the underlying substance” --Wikipedia

During and after WWII, some native south pacific islanders erroneously associated the presence of war related technology with the delivery of highly desirable cargo. When the war ended and the cargo stopped showing up, they built crude facsimiles of runways, control towers, and airplanes in the belief that the presence of war technology caused the delivery of desirable cargo. From our point of view, it looks pretty amusing to see people build fake airplanes, runways and control towers  and wait for cargo to fall from the sky.

The question is, how amusing are we?

We have cargo cult science^[1], cargo cult management^[2], cargo cult programming^[3], how about cargo cult system management?

Here’s some common system administration failures that might be ‘cargo cult’:

Failing to understand the difference between necessary and sufficient. A daily backup is necessary, but it may not be sufficient to meet RPO and RTO requirements.

Failing to understand the difference between causation and correlation.^[4] Event A may have caused Event B, or some third event may have caused A and B, or the two events may be unrelated and coincidental.

Failing to understand the difference between cause and effect.

Following a security recipe without understanding the risks you are addressing.  If you don't understand how hackers infiltrate your systems and ex-filtrate your data, then your DLP, Firewalls, IDS, SEIM, etc. are cargo cult. You've built the superficial exterior of a system without understanding the underlying substance. If you do understand how your systems get infiltrated, then you'll probably consider simple controls like database and file system permissions and auditing as important as expensive, complex packaged products.

Asserting that [Technology O] or [Platform L] or [Methodology A] is inherently superior to all others and blindly applying it to all problems. When you make such claims, are you applying science or religion?

Systematic troubleshooting is one of the hardest parts of system management and often the first to 'go cargo'. Here’s some examples:

Treating symptoms, not causes. A reboot will not solve your problem. It may make the problem go away for a while, but your problem still exists. You've addressed the symptom of the problem (memory fragmentation, for example), not the cause of the problem (a memory leak, for example).

Troubleshooting without a working hypothesis.

Changing more than one thing at a time while troubleshooting. If you make six changes and the problem went away, how will you determine root cause? Or worse, which of the six changes will cause new problems at a future date?

Making random changes while troubleshooting. Suppose you have a problem with an (application|operating system|database) and you hypothesize that changing a parameter will resolve the problem, so you change the parameter. If the problem reoccurs your hypothesis was wrong, right?

Troubleshooting without measurements or data.

Troubleshooting without being able to recreate the problem.

Troubleshooting application performance without a benchmark to compare performance against. If you don’t know what’s normal, how do you know what’s not normal?

Blaming the (network|firewall|storage) without analysis or hypothesis that points to either. One of our application vendors insisted that the 10mbps of traffic on a 100mbps interface was the cause of the slow application, and we needed to upgrade to GigE. We upgraded it (overnight), just to shut them up. Of course it didn't help. Their app was broke.

Blaming the user or the customer, without an analysis or hypothesis that points to them as the root cause. A better plan would be actually find the problem and fix it.

Declaring that the problem is fixed without determining the root cause. If you don't know the root cause, but the problem appears to have gone away, you haven't solved the problem, you've only observed that the problem went away. Don't worry, it'll come back, just after you’ve written an e-mail to management describing how you’ve “fixed” the problem.

It's easy to fall into cargo cult mode.

Just re-boot it, it'll be fine.

---

^[1] Richard Fenymen, CARGO CULT SCIENCE: http://www.lhup.edu/~DSIMANEK/cargocul.htm
^[2] Mike Speiser, Cargo Cult Managment: http://gigaom.com/2009/06/21/cargo-cult-management/
^[3] Wikipedia: Cargo Cult Programming: http://en.wikipedia.org/wiki/Cargo_cult_programming
^[4] L. KIP WHEELER, Correlation and Causation: http://cnweb.cn.edu/kwheeler/logic_causation.html