Thursday, August 19, 2021

Quantum physics, coordination and removing impediments

 Not everything is a "project", but there are other tasks that a PjM can do to move things along.

Some recent observations:

Observer Effect 

Sometimes a "job"needs to be done, but is stalled. Simply assigning a PjM to the job can get  the ball rolling.

Recently I was assigned to "get this job done". I had no idea what it was, but by the time I had gathered together a team on Slack, and the team added a few more necessary members, the task was done.

Simply getting the correct people to talk to each other was 90% of the job.

A while ago I took an orphan task that had started way before I started at the company. By simply asking where it's holding and what needs to be done, gave the actors the incentive to finish the task and roll it out.

Unclear what's at work here; maybe it's the Observer Effect - "Physicists have found that observation of quantum phenomena can actually change the measured results of this experiment."


Sometimes a job - or even a project - requires multiple teams to work in serial. The handover from one team to the next is often stalled until a PjM steps up and makes it happen.

This is less apparent - and less needed - when teams work together; the coordination is built in at design time and during the kick-off stage.

Removing impediments

Removing impediments is another PjM speciality. A team may be stuck because they are missing equipment, or a 3rd party license or an outside contractor. 

A PjM sometimes needs to step in, discover what is missing and then remove the impediment: go to finance to get the requirement purchased, or the CTO to get permission to hire a contractor. Barely Project management, but very necessary for getting things done.

Tuesday, August 17, 2021

"A meeting without minutes is a kumzitz"

 "A meeting without minutes is a kumzitz"

                                    - Danny Schoemann

What do you think of my new philosophy? How useful are meetings if nobody keeps track of what is being discussed, what the action items come up and without follow-up from the previous meeting?

Even when schedules and Kanbans are being updated in real-time during meetings, I still believe that meeting minutes are important, for example for those who aren't present or for historical perspective.

Sunday, May 9, 2021

Solving Ransomware attacks with Project Management

Ransomware attack leads to shutdown of major U.S. pipeline

That was the headline that caught my attention this morning.

As a Project Manager, certain thoughts  crossed my mind - using subconscious techniques like risk management, crisis control and borderline-use-cases:

1. All ransomware attackers want to be paid in cryptocurrency. 

The whole world is suffering  from a crackdown on grey & black-market currency. In Israel - for example - there's a limit as to the amount of cash you can pay when buying items. How come you have so much cash?

Every time I transfer money to one of my kids, I worry that some algorithm will flag it for some bored paper-pusher to investigate, and waste hours & days (and possibly lawyer fees) to prove my innocence. 

Yet, you can squirrel away your criminal earnings into crypto and voila! life is good. No way to trace what you do with it. No way to trace you are you.

Clearly, it's time to change the anonymous aspect of crypto. (I'm a firm believer that only people with things to hide refuse to divulge information that would help solve international identity issues. From biometric identity documents to cryptocurrency, and more. But that's not the point I'm dealing with here.)

I'm not sure how to go about it, and I sure hope that law-enforcement agencies are working on this...

2. It's not the mouse, it's the hole that steals.

A somewhat amusing Talmudic statement - shifting the blame from the mouse (or hacker, in our case) to the hole.

So, how do we fix the hole? That's the cat & mouse game (pun intended) that half the world is playing against the hackers. 

But, if one can mitigate the affects, to the point where one doesn't even consider paying the ransom, one has shrunk the hole tremendously.

Here are some ideas for mitigating the pain of a ransomware attack, based on being responsible, inter alia, for IT and DevOps for the past 9 years.

A. Backups

Duh. Don't you have anything to write about? Well, let's reword that to backups and fast restores. 

I refuse to believe that something as crucial as a U.S. gas pipeline supplying the entire East Coast does not have backups in place. But... if they cannot restore in a timely fashion, then they will have to work with a crippled system for a while.

This is unacceptable. The smallest Mom & Pop shop will suffer from a few days down-time. The more critical the computer system, the less it can afford down-time. So it's crucial that the backups are:

  • Actually happening  - and not suffering from out-of-space or limited bandwidth issues, or other excuses.
  • Up to date - you don't want to lose more than a few hours (or minutes) of work. 
  • Fast restore - the closer to instant restore, the better.
  • Ransomware-safe. If the backups are hacked, or you are backing up a system with the ransomware attack installed, you essentially don't have a useable backup.

Let's discuss these last 2 points in more detail.

Fast restore

As already mentioned, if you cannot restore in a timely fashion, you are going to be without the ability to function; from being unable to sell lollipops to having patients dying.

One technique I came across was to virtualize everything - then the machines can be backed up periodically by taking a snapshot. This technique is used on VMWare servers, for example, and is easy to automate, doesn't require downtime (you can backup while working) and the restore is almost instant.

I've also  used offline-continuous-backups like Carbonite; the restore can take a while, depending on your bandwidth. Your apps won't be restored - you'll have to reinstall them -  but your data will be almost up-to-date.

Ransomware safe

You have to ensure that you have a backup from before the system became infected with ransomware. Otherwise, the ransomware will attack again once everything is fully installed.

A rotation of 7-daily and a few weekly backups is one way to ensure you will have a clean restore point. Yes, it will take up a LOT of disk space; but, hey, disk space is almost free nowadays. Compare $800 for a 20TB disk to 1 BTC (1 Bitcoin is North of US$57,000 - yes, fifty-seven-thousand dollars).

That's 70 external disks of 20TB each.

Make sure you have on-site copies - for quick restores - but make sure they are off-line, or else ransomware will destroy them too. Yes, this requires frequent manual intervention, but is well worth it.

Do yourself a favour and keep off-site copies (like in AWS S3) for off-site security. (Good in case if fires, thefts or ransomware to note a few possibilities.)

Mirror, mirror on the wall

After many decades in Hi-Tech, few work-relate incidents stand out in my memory as starkly as those days that I got to work and my computer was dead.

Walk over to Tech Support (since you cannot email without a computer - and in the olden days we didn't have smartphones; they hadn't been invented yet) and after hours of walking around aimlessly you can start working again. 

Well, not really, you have to reconstruct your work environment, reinstall your favorite apps, map your favorite drives, etc.

All this changed about 9 years ago when I became responsible for IT. Who does IT go to when they need IT? Who cuts the barber's hair? Who does a dentist go to if he has a toothache?

What I did was to take an older machine and - one that was too old & slow for normal work - and install everything I needed on it. Every time I installed a new program I would install it on my mirror-machine too.
The rest of the time the mirror-machine was powered off. (This created an interesting question when we were audited for licenses.)

When the dreaded day arrived, I had to deal with reviving my laptop . But I could continue working - I just powered up my mirror-machine and everything was there, as I liked it.

This got me thinking. Why don't hospitals, government offices and everybody else  who has mission critical  computers have a mirror system in place?

 Then when the ransomware attack happens, or flood, or fire or theft, everybody uses their mirror-machines. Of course, you would need to be careful not to infect the mirrors - possibly having them on their own network - and maybe in a separate room/building.

The devil in the in the details, but I want to raise the awareness of these possibilities.

Mix & Match

Now imagine if you would virtualize all your machines and restore them frequently to mirror-machines.
Then in an emergency you would have an almost up-to-date useable system, with minor down time.


Thinking though the details and what I should include to keep this a manageable size, I realized that I may have a possibility for a business.

Anybody want to join me? Creating a system implement and automate the mirror-backup system? Or maybe off-site locations to instantly relocate to in times of system failure?

I no longer want to start the day with headlines about impeding catastrophe because of ransomware.

Tuesday, November 17, 2020

An amazing discovery - the 12 Factor App

 I have a daily reminder that pops up and reminds me to write something on this blog.

As you can see, I have not been persuaded nor inspired nor had enough free time to do so.

But today I discovered The Twelve Factor App - and I'm excited enough to want to share it - and document it so that I don't lose track of it.

For example, it seems that I'm not the only one who treats things like Version Control as a crucial part of development. 

Yes, I know, some of you will be aghast at the thought that this isn't always so.  Others will wonder what that is. 

12 years ago I promised that "In a future post we'll discuss why you never want to allow your project to proceed without some form of Version Control" - looks like I'm in good company.

In a nutshell, the 12 factors - copied from their site - are:

I. Codebase

One codebase tracked in revision control, many deploys

II. Dependencies

Explicitly declare and isolate dependencies

III. Config

Store config in the environment

IV. Backing services

Treat backing services as attached resources

V. Build, release, run

Strictly separate build and run stages

VI. Processes

Execute the app as one or more stateless processes

VII. Port binding

Export services via port binding

VIII. Concurrency

Scale out via the process model

IX. Disposability

Maximize robustness with fast startup and graceful shutdown

X. Dev/prod parity

Keep development, staging, and production as similar as possible

XI. Logs

Treat logs as event streams

XII. Admin processes

Run admin/management tasks as one-off processes

Of course, the devil is in the details; I plan on studying how they see each of these 12 steps - and commenting on them. Stay tuned.

Wednesday, November 14, 2018

Project management has changed!

 "Project management has changed" is what I was informed.

As a result, this head-hunter refused to consider me for a PjM job; since I've been doing 1,000 other things for the past 7 years.

The fact that I've been sharpening my technical skills and updating my knowledge of the Software Development world is irrelevant.

So, my questions are:
  • Has PjM changed, and how?
    • I don't recall noticing any new fads, even though I frequent sites like almost daily, and I am ranked in the top 12% of 23,000+ users.
  • Does it really matter that I haven't done any active PjM work for a while?
    • Keep in mind that I never formally learned PjM, it's a skill I'm born with.
  • Shouldn't it be advantageous that I have hands-on experience with state-of-the-art technology?
    • Or do they prefer PjMs who haven't had a chance to keep up with the latest technologies?

Your thoughts are appreciated. My thoughts on the subject were written 6 years ago at the post titled
Why a technical Project Manager?

    profile for Danny Schoemann at Project Management Stack Exchange, Q&A for project managers

    profile for Danny Schoemann on Stack Exchange, a network of free, community-driven Q&A sites

    Wednesday, November 16, 2016

    Do projects ever end early?

    For a while I was an active participant in the StackExchange Project Management beta site.

    One of the 6 questions I asked was Do projects ever end early?

    The fascinating part of this was that it has been viewed about 6,400 times and generated 14 answers! 
    (That's a lot of traffic for a site that has been in beta for almost 6 years because it averages a mere 1.5 questions per day. It needs 10 new questions per day to get out of beta.)

    It seems that a lot of people are wondering the same thing: Do projects ever end early?

    The question I posed was:
    When scheduling, I tend to add a lot of reality into the guesstimates I am provided with.
    I always add in generous amounts of probable sick days for manpower and extra time for integration and bug fixing. (All this based on decades of experience.)
    As a result, my projects tend to deliver on schedule. But they never finish earlier than expected.
    Now I'm wondering if the projects would deliver earlier if I didn't pad the schedule as much.
    Would everything move a little faster if the teams had the incentive to meet deadlines based on the data they provided, as opposed to the padded ones I provide?
    Or put another way: Does work expand to fill up available time?

    To see the diverse answers, you'll have to go to the site - way too much to paste here.

    But the bottom line remains that with a realistic schedule you could finish on time and possibly early.

    A realistic schedule is not based on when the customer (or boss, or calendar, etc.) wants it done.

    A realistic schedule is created by finalising the spec and then having an experienced Technical Project Manager create a schedule based on input from those doing the work. This schedule will include time for integration, testing and sick leave, amongst other things.

    Note that I wrote based on input from those doing the work. An experienced and Technical Project Manager will have a pretty good feel how to take those guesstimates and morph them into a schedule.

    E.g.: An experienced and Technical Project Manager will know that people do not realistically work 9 full hours a day, and that no task can take less than a few hours. More about that another time.

    Monday, January 16, 2012

    Managing without management

    For the past 6 weeks I have been working at a new company - hence the lack of updates on this blog.

    My previous employer - - was big into management and SEO.

    My current employer - Hyperlync - does not believe in either.

    Their site is all but non-existant - and most pages are carefully spider-proofed. They generate revenue by creating products that people actually use, and they have no need to be in Google's index. Those people who need their products know where to find them.

    This means that we never hear - and really don't care - about Google events and other tragedies that affect SEO-driven companies. What a relief.

    They also don't believe in management. The 3 founders all work for a living; 2 of them write code the 3rd does the sales & marketing.

    There are no status reports and nobody looking over your shoulder. As soon as a product is sold, the relevant employees are informed and each does his part to make it ship.

    Since there are no teams, you cannot rely (or hide behind) your teammates or superior to get the job done.

    While this may sound like a startup, it's not - and they are already profitable; so much so that we each got the latest version Kindle to celebrate Chanuka / 2012.

    There is one Project Manager, but she mainly does product-management; deciding on the look-&-feel. She does almost no coordinating, scheduling or running around.

    More on management-less companies in future posts.

    - Danny

    P.S. So what do I do? I went back to 1995; doing setups. This time on Linux, Mac, Windows and Androids.

    Sunday, November 27, 2011

    Bottlenecks: Never assume the experts know better

    As the saying goes: "Experts built the TitanicAmateurs built the Ark."

    It seems to be human nature to rely on experts and not question their decisions. After all, if they are so clever, who are we to question them?

    But sometimes it seems that they are self declared experts and/or it is us who declare them as experts and then believe our own prophecies.

    Jerusalem has 2 new fast transport systems: A light rail and express buses. The aim is to eventually replace the latter with another line or 2 of trams.

    I always wondered what was the advantage of transport-by-rail - especially intra-city where speed is not an issue.

    Light rails are great when they work; the idea is that cars and pedestrians have to keep out of their way, as the trams are inflexible. They are quiet and don't need to tank up as they are electric.

    Two weeks ago I saw that anything travelling on rails suffers from the typical bottleneck problem, as to be expected.

    First I saw a train that had started pulling out the station and got stuck; I have no idea what precisely happened. For about 20 minutes it simply stood there. It was jam-packed and nobody was allowed off. Eventually the driver came around to the cabin at the back of the rain and drove it back to the station and let everybody off.

    So what happens to the other trains? They have no way around the stuck train, so essentially all transport  heading North was stopped. It was a matter of time until all South bound traffic followed, obviously. (There are a few places where the North and South bound rails have an option to cross over; but running the trains in both directions on the same rail doesn't sound like a great idea.)

    Later that day was one of the biggest funerals that Jerusalem has seen. It was close to the rail line. It brought the train system to its knees - for the second time in the same day. Since everybody wanted to get on the train, the train got stuck at every station, as the doors don't close if people are in the way. I sat watching the electronic devices inform us the the next train was in 10 minutes, then 15 minutes and then.... it announced "train stopped". It took 45 minutes for the train to arrive.

    Despite them announcing that the next train was right behind (and probably 3 more behind that one) everybody tried to get on this one!

    Buses are way more efficient! I often seen 3 buses play leap frog; one fills up at a bus stop while its twins head for the next stops.

    Even in the express bus lanes, a bus will overtake another, if the first bus is stuck or taking too long to move.

    There is a way to get both the advantages of quiet electric transport and not have to deal with rails.

    In Johannesburg they used to have (and probably still do, but I have not been there in over 20 years) double decker electric buses with 2 antennas connected to the overhead electric lines. If a bus got stuck it could be unhitched from the wires and other buses could get by. 

    They also had to do load balancing; if too many buses were behind each other trying to get up some of the steeper hills, then they all would get stuck unless some of them stopped, giving the power back to the ones in front.

    Maybe the light rail was not such a brilliant idea after all...

    Thursday, November 3, 2011

    Are all changes really improvements?

    I can just imagine it: The yearly review at Egged; the Jerusalem bus cooperative. The major request for change must have been to get rid of the need to punch those paper bus tickets.

    It was slow, it meant interacting with each passenger. it meant that the person driving a million-dollar bus spent a good part of his day punching holes into paper; paper handled by other people and he had no way to wash his hands.

    So, many years ago they created the monthly bus pass. Now the bulk of travelers simply flash their passes and walk by. What a major improvement.

    In parallel they decided to upgrade to the 21st century and computerize the system.

    Welcome to CityPass. Each passenger has a debit card - this should really improve things.

    But, as often happens, this was not thought through carefully. Now every single passenger needs to insert their bus card into the machine; the driver punches a button and - after the light on the machine turns from red to green - you can remove your card and the line behind you can advance slightly.

    As an added bonus, at major bus stops (like the Central Bus Station) you can no longer enter by the back door after an inspector punches your card.

    Net result: Travel time by bus has increased noticeably.

    A spec-review would have been in order here! A project manager would have picked up on issues like:

    - The machine which validates the cards better be lightning fast if the driver is to deal with each passenger

    - Alternatively there could be card readers at every door so that the driver does not have to deal with any passengers. This is how the light rail has it planned out. On 1 Dec we shall find out if their system really works.

    A few simulations would have picked up these issues.

    Now I'm waiting to see what happens as the magnetic strip on the cards start be wear out... that will surely wreak havoc on the system as people alight and discover their bus cards are dead...

    Conclusion: Not every change is an improvement.

    Sunday, October 30, 2011

    Spec review: Airbags?

    One of the many skills a Project Manager needs, is the ability to review functional specifications.

    A Project Manager needs to be able to run a spec-review meeting and follow up the action items and document the changes resulting from the meeting.

    Another required skill is the ability to persuade people that everything needs a spec and a review.

    Why is that? Let's take a non-programming example, for a change; since I've been getting complaints that my blog is very programming oriented.

    In 1971 GM added airbags to the driver's side. Since the airbag worked well for the driver, why not install it on the passenger's side also. Source: Wikipedia: Airbags.

    Sounds simple enough: Here's what the MRD (Marketing Requirement Document) may have said: "Take existing design of airbag and add it to the passenger side."

    Hardly needs a design, functional spec and review, does it? Seems that implementing it will be trivial; what could possibly go wrong?

    If anything, it's way more complicated to add it to the driver's side, since the steering wheel is in the way.

    Fortunately they did a spec review, and here's some of the issues that were revealed:

    • What happens of an infant is in the passenger seat? Won't the airbag serous injure / kill it?
      • Solution: Put a weight sensor. Do not deploy if weight less that 3 year old child. 
      • Action item: Find out minimum weight from medical dept.
      • Action item: Inform legal that a new "anti-infant" clause needs to be added to standard car contract.
    • What happens if the infant is in a car seat? it will be heavy enough to be detected as an adult... yet the result could be fatal...
      • Add switch to manually override the airbag.
        • Where should we place this switch? passenger side? drivers' side? in between?
          • Since it's to protect infants, needs to be on driver's side.
          • Needs to be far enough away from passenger that kids don't play with it during travel.
        • Does it reset when ignition is turned off, or does it remember its state?
          • Could be fatal if it resets; better if it remembers state.
        • Add a warning beep to sound when airbag is turned off.
        • Add a warning beep if passenger is too light and airbag is turned on.
        • How can we prevent it being turned off accidentally?
          • Maybe it should require car-key to flip on/off.
          • Decisions: try both options and track customer feedback.
      • Add light to dashboard to indicate that airbag on passenger side is deactivated.
      • Action item: Design relevant icon - graphics dept.
    • How will drivers know when to turn the airbag off, and what all the icons and beeping mean?
      • Action item: Get stickers made to explain the infant-hazard; get wording from legal, design from graphics dept.
    • Should airbag always deploy?
      • Maybe it should only deploy if the seat-belt is buckled? This way it won't unnecessarily if only groceries are in the passenger seat.
      • On the other hand, the airbag is most useful when the passenger is not wearing their seat-belt.
        • Decision: Deploy independently of seat-belt being buckled.
    As we can see, this trivial, easy to implement feature has become a project involving the following departments: legal, graphics and medical.

    It requires a redesign of the buttons sections and the icon on the dashboard. It also requires a new type of switch that gets turned by the key.

    We now know why the first passenger-side airbags didn't appear until 1973.

    Finally, we also have plenty information in order to create a full-length functional specification, which everybody will understand why it's needed, and many people from various departments will be required to review.  

    So now we know!

    - Danny Schoemann