On course computer totalisators
Courtesy of Brian Conlon:
The new totaliastors serviced Eagle Farm gallops, Doomben gallops, Albion Park trots, The Gabba greyhounds and Bundamba gallops. These systems were commissioned in 1979. The transaction processors were PDP11 mini computer based and the ticket issuing machines were J22s which utilised the M6800 microprocessor as a controller. These computer totalisator systems superseded electromechanical Julius totes which had been operating for decades at each of the locations mentioned above. The company Automatic Totalisators was founded by George Julius, later Sir George Julius, who invented the world’s first automatic totalisator here in Australia in 1913. The Julius totes at the Brisbane tracks which were superseded were descendents of this original invention.
Following are some recollections related to the introduction, operation and maintenance of the new computer totes as well as information on the Julius and Hodsdon totes. They concentrate on events that either contrast with the ways of today, which tends to make it more interesting to contemplate how different the times were, or that have a curious or humorous element.
The Brisbane Project which was established to develop the new totaliastor systems, which were a new tote product line for Automatic Totalisators, existed in the shadow of a much larger project Sha Tin. As a result, in my opinion the Brisbane Project did not receive the resources that it should have. When the Sha Tin project failed due to inability to deliver on time, the success of the Brisbane Project became essential to the continuation of the company as all future contracts had a qualification clause stating “Dependent on the success of the Brisbane Project”. Massive overtime was worked by the handful of employees who were familiar with the Brisbane Project to ensure it successfully opened on time.
This overtime resulting in our presence on the racetracks late at night, conflicted with the lifestyle of the Racecourse Managers, which required rising early in the morning and consequently retiring early at night. A sequence of events regularly occurred at one of our Brisbane tracks however these circumstances generally applied to all the tracks. The Racecourse Managers usually had guard dogs, which fortunately, whenever I saw them, were in the presence of their owners, who quelled the dogs’ evidently overwhelming desire to sink their teeth into anyone they did not know. To continue this anecdote, I must point out that the Brisbane Project provided two totalisator systems each housed in a semi trailer which would move from track to track and that we had no keys to the tracks. Around 8PM the Racecourse Manager would inform us that he was finishing for the night and letting the guard dogs out. Fortunately most of the work was on the transaction processors inside the vans and we generally remained inside until it was time to leave. We developed a procedure to exit the track in which we became particularly adept. We would open the large trailer door a crack to peer out to see if there was any sign of the dogs. Having established the coast was clear we would open the door and exit, close and lock the door, as rapidly as possible, keeping noise to a minimum, as the door was heavy with a heavy locking mechanism, which tended to clank and squeak. We would then dash as silently as possible to the nearest point of the outside fence. One would clamber to the top of the fence and the other would pass up the brief cases one at a time to the person at the top who would drop them on the outside of the track. The one on the ground would then clamber up on the fence as quickly as possible to minimize the opportunity for the dogs to catch him. I recall many occasions sitting on top of this fence around 2AM relieved that we had successfully run the gauntlet once more and thinking, whilst preparing to jump down on the outside of the track, that when I entered the computer industry I never envisaged this would be the method of exiting work. I have often since wondered what the present day OH&S fraternity would have had to say if confronted with this procedure.
One of the Brisbane computer trailers with one of the standby power generator trailers. Photograph by Kevin Egan
The Brisbane Project introduced two major milestones to the product line of the company. Firstly, it was the first sell pay system it produced. All the predecessor systems including the computer totalisators were sell only requiring manual payment of dividends. The new system generated bar codes on the tickets which were machine readable. The system was capable of calculating the dividends which were referenced after the TIM (Ticket Issuing Machine) read the ticket to determine the payment value which was reported to the operator. In the early years of this computer tote, these dividends which were calculated by the on course tote were disseminated throughout the state via an inter-cpu link with QTAB (The Queensland TAB) as it was then known. The second milestone of the Brisbane System was that it was the first where the transaction processors utilised an operating system which in this case was the Digital Equipment Corporation’s RSX11M resource sharing multi user operating system. More can be read about the new computer totes at the Totalisator History website at http://members.ozemail.com.au/~bconlon in the chapter titled “ATL The Brisbane Project.”
Brisbane was reminiscent of a large country town when this system was being commissioned. At times when the on-site project team finished early, before midnight, and wanted to spend some social time together, finding a place to eat and have a drink after 10PM seemed impossible. Our Engineering Manager ended up insisting on staying in one of the very few CBD hotels which had 24 hour reception and room service. It was in his room that we congregated on these occasions. His ability to lean back in a chair with his hands holding a glass of beer on his chest and catch forty winks without spilling a drop was famous. This Engineering Manager led by example and was a crucial element in the success of this project.
The opening of the new on course totalisator systems in Brisbane was my first experience with on line totalisator operations. The chief programmer used to pensively pace up and down in front of the computer racks during operations. His ability to know when to start frantically typing MCR commands (the command language of the RSX11M operating system) on the system console to rectify the problem before the phone calls indicating something was wrong amazed me. This amazement developed into intrigue when I realised this ability extended to anywhere on the track. Work was so busy that, although we spent masses amounts of time together at work I never had the time to discuss it with him. I later discovered that the time had come to intervene whilst in the computer room was when the disk drives fell silent. These drives do not match the image that a disk drive of today conjures up. Although small for the time as a result of utilising a new recording method MFM (Modified Frequency Modulation) giving a packing density of 5600 BPI (Bits Per Inch) similar capacity drives of the time, were the size of washing machines. The drives the chief programmer was listening to only weighed 65 pounds (29Kg). The head positioner arm and motor were large and heavy and made a clanking sound whilst seeking. When this fell silent, it was an indication that the transaction file was no longer being updated and implied that betting had stopped.
The PDP11 tote transaction processing system inside one of the computer trailers. Photograph by myself Brian Conlon
Out on the track it was the length of the queues that gauged the performance of the system. He was accustomed to the normal queue lengths and if the lengths grew beyond normal it was time to race back to the computer systems. Oh and to put this physically large and heavy by today’s standard disk drive into capacity context, the storage capacity of these drives was 50MB and the drives available nowadays in 2010 are up to 1 to 2TB making the 2TB drives almost 42,000 times the capacity of the late 1970s drive. To match this capacity with the 1970s drive you would need 41,944 old drives which would weigh 1,216,376Kg to equal the capacity of a 2TB drive that can be held in the palm of your hand.
Following is a, humorous in retrospect, look at a sort of highly risky procedure one is keen to ensure is avoided in real time computer operations. The Chief Engineer of the electromechanical systems was removing his jumper in the operations end of one of the computer vans. He removed one arm from his jumper then swung the jumper around his back to remove the other arm. As the jumper passed one of the control terminals a loud crack was emitted resulting from the static discharge. After a few blank looks at each other, we discovered the application running on this terminal had hung and was locked in memory. This was a problem in the days of sparse memory resources. Physical memory was 124KW (Kilo Words) with a 4KW I/O page. It was essential to retrieve the now unusable memory allocated to this application. An abort command would sometimes not run to completion leaving the application marked for abort without the abort running to completion. In this event it was time to resort to Open, a system utility program for examining and altering the contents of main memory corresponding to addresses where the operating system resided. This procedure required the use of the task builder map to locate the absolute address of the start of the system task control blocks. An offset was then added to this address to point to the beginning of the task control block of the application in question. Another offset was added which produced the address of a status word in this block. Now one bit had to be set in one byte of the word leaving the high byte in the word along with the other bits in the low byte unaltered. If you got this right the program would abort. The implications of getting it wrong were some form of erroneous system behaviour or in the worst case scenario a system crash. Having mentioned the 124KW of physical memory it is interesting to compare this to the common 3GB of home computer main memory today. There are 2 bytes in a PDP11 word giving 248KB requiring 20,529 PDP11/34 main memories. This is not quite the complete picture as the main memory of the PDP11/34s in the computer totes used 16KW core memory stacks to begin with, which had a PCB (Printed Circuit Board) approximately the size of a keyboard, not quite as wide and a bit taller. There were 8 of these to provide the 124K words of main memory requiring 164,232 of these boards to match the 2GB memory. For the technocrats reading this, the top 4KW of main memory was disabled to make room for the I/O page addresses and this has not been accounted for in the calculations to keep it simple.
At one of the early operations during a hot day, it became apparent that the computer air conditioning was struggling and allowing the temperature to rise. I explained thermal runaway to our branch manager and indicated that this is a fatal condition for semiconductors, which are used to implement the combinational and sequential logic, which constitutes the major building blocks of digital computer systems. The tote manager immediately started running around waving his arms in the air, shouting "HOSE it down! HOSE it down!...” I followed in a similar manner protesting “No! No! Anything but the water!” This raises an interesting point, that this era well and truly predated widespread general knowledge of computers and of electronics in general for that matter. People who were being trained to operate these systems were apprehensive working with a computer in the first place and then these systems, which were cantankerous at the best of times, due to hardware and software problems, gave them good reason to become anxious. This history also predates the advent of home computing which was the most significant influence elevating general knowledge of computers. I recall that when I wanted to type a memo at home for work, I had to use a typewriter. For people who were outside the computer industry and fortunate enough to have seen one it was usually a mainframe class machine they had seen. These used to reside at large company headquarters inside security restricted computer rooms which sometimes allowed the inside to be viewed through a window and non computer staff could peer through at the behemoth and see computer room staff performing clandestine duties operating them.
On the subject of anxiety experienced by the operators, one of the Race Day Controllers was a Catholic Priest. He was very cool under pressure, which this system was not short on providing and he had a calming influence on the other Race Day Controllers. He was the most broadminded priest I have ever encountered. He was a delight to talk to and you could even discuss subjects like reincarnation and the possibility of extraterrestrial life without being regarded as a heretic. And this was over 3 decades ago with a lot of new age tolerance and even acceptance of such subjects taken place since. Whilst contemplating the inconsistency of a priest in a gambling related industry, consider the fact that George Julius’ father Churchill was Anglican Archbishop of New Zealand. Churchill is said to have been a religious liberal. George claimed that his mechanical abilities were inherited from his father.
Utilising Minicomputer systems in a mobile application was an unheard of concept, which had technologists looking at you in disbelief when telling them of the mobile totaliastors based on PDP11s. There were “ruggedised” versions of these computers which I believe were mainly for military applications however we were not using them. This was so unheard of at the time that this may have been another Australian first! In practice the use of these minicomputers in a mobile application was not without significant problems. Minicomputers were very hardware intensive in comparison to the Microcomputer class machines. As previously mentioned this history predates home computing and it was the Microcomputer class of machine that would become the basis of the home computer industry. At the time their major application was in device controllers like the M6800 microprocessor in the J22 ticket issuing machines mentioned. Also as previously mentioned these totalisator systems were cantankerous, with no shortage of hardware failures, not to do with their manufacture, but related to the mobile application and the associated vibration. An additional contributory factor was opening with a system that was in need of additional software development and debugging, as a delay was not possible at the time and it had to open “now or never”. Finally add to this air conditioning that struggled during the hotter summer days triggering thermal runaway failures in semiconductor devices, the building blocks of electronic systems and you end up with major problems all contributing to the anxiety, previously mentioned, of staff working with these systems.
It is not possible to consider that the maintenance demands of a system that taxed the efforts of a 9 man technical team to the extreme at times, could have gotten any worse, however then there was the Jinx! Ironically he was one of our most beloved of our managers. This new tote system, although quick to present problems at the best of times, excelled itself in its misbehavior when he visited. This Jinx effect was very clear and continued to demonstrate its effects for the lifetime of the system. He noticed the impact he was having and resolved to stay at the other end of the track when visiting. I found this not to be particularly effective as the system seemed to know he was on the premises. The final crescendo of these strange events occurred years after he retired. We had a particularly bad day one Saturday at Eagle Farm. We had a tote system operating at Caloundra. One of the duplex transaction processors operating there failed requiring a fallback and repair of a failed disk drive. In addition the Eagle Farm system had an inter-CPU link with QTAB and this link was experiencing significant problems which we had to rectify. Finally, a disk drive in one of the Eagle Farm tote system’s duplex transaction processors had a spectacular failure. The power amplifier board, which drove the positioner motor burst into flames. When all the problems were quelled, my 2IC and I staggered out of the computer trailer, stepping onto the mobile landing which had an attached staircase and exclaimed, “One could be forgiven for thinking”, and before I could get the rest of the sentence out which would have been “Bruce was here”, I looked down the stairs and saw him standing there. He had come to visit and see how we were all getting along! As a result of the cantankerous nature of this system, despite significant improvements in the system over the years, in the near decade that this computer totalisator system operated, we never went to work free of the thought “What is it going to do to us today”.
Another aspect of this system that has just about disappeared is the maintenance philosophy applied to it. Everything was repaired to component level. The reason for this, the most demanding of the maintenance methods, was that the systems were too expensive to purchase spare ones and implement module level maintenance. Also for some of the equipment, if it was not repaired by us, it had to be returned to the USA for repair and the delays involved were intolerable. On the software side of things, efficiency of the code written for this system was of prime importance considering the sparse main memory resource. Significant use was made of assembler language for this reason. The rest of this tote system was written in Fortran or Ratfor, a rational form of Fortran. The importance of efficiency of code has declined over the years with the massive increase of main memory resources and processor performance. For more information on computer tote maintenance see http://members.ozemail.com.au/~bconlon and select the computer Tote Maintenance chapter.
Today in the era of mobile phones, one feels disconnected leaving home without one and thinking of a time when there were no mobile phones seems archaic, let alone contemplating having no telephone at all. This was the case when this computer tote system started and continued for some years as it was illegal to have any telephone on course. When we used to be at the racetrack late at night, trying to rectify an end of session procedure which had gone wrong, or repairing a system so it would be available to run the next meeting, which were regular events, it was not possible to quickly ring home to say that we were still alive and well. As we were locked inside the track, entering and exiting at night required climbing fences so it was easier not to call. One night the wife of one of our technicians had enough of the anxious waiting for him to come home and rang the police. The police arrived at the trotting track where the technician happened to be working that night, and awoke the racecourse manager there to enquire about the technician’s whereabouts. This caused a political issue resulting in our management stating that they did not wish to see a repeat performance however I felt that she had made a good point. The long working hours was evidently nothing new. Charlie Barton who was the Chief Engineer of the electromechanical systems in Brisbane, had a term for wives of tote engineers, he called them tote widows.
I was responsible for the engineering side of the new computer tote systems and inherited the technical staff that had been working on the Julius electromechanical totes for a very long time. I inducted them into the computer tote era and I marvel at the achievements of some of those staff members who had such a large technological gap to jump. The magnitude of this technological gap was never as apparent as on one weekend, when I supervised and assisted with some work on an old electromechanical system. This was my one and only experience with a live electro mechanical system. This was a relatively modern one, part of which was based on Uniselectors. One of the technical staff, mechanics as they were called then, indicated that the presence of a DC voltage at a TIM (Ticket Issuing machine) had to be checked. As we walked to the TIM I noticed that the only tool we had between us was a rather large screwdriver my companion was carrying. I presumed he had left his meter for measuring the voltage near the TIM and excluded the possibility that we would be using a CRO (Cathode Ray Oscilloscope) to be performing the measurement. When we arrived, the lid of the TIM was promptly removed and the screwdriver was inserted inside the machine promptly followed by an audible spark. My immediate reaction was "well that's torn it" – Australian translation “it’s stuffed”. The look of satisfaction on my companions face took several seconds for me to comprehend: The test was successful! I concluded that this was a satisfactory test when working on these systems, short circuit a voltage source and if it sparks it is working. With the computer system’s hardware you quickly learnt to avoid any possibility of causing short circuits. The resulting damage would be quick to inflict leaving what sometimes could be lengthy lamentation time as you repaired the results of your negligence.
Having introduced the Julius totes I will provide a little information on them. George Julius invented the world’s first automatic totalisator here in Australia in 1913. That system was installed at Ellerslie in New Zealand and operated there as did subsequent Julius totes. George later Sir George, knighted for his contribution to Australian technology, founded the Australian company Automatic Totalisators to further develop and export these systems. This Australian company became a world monopoly in this field in its early years. The Julius totes dominated the totalisator industry for two decades and then became part of an oligopoly when competitor manufacture of electromechanical totes began in the 1930s. The oligopoly remained until the advent of the digital computer when the business became openly competitive. Julius totes were installed in Australia, New Zealand, Singapore and other locations in South East Asia, India Ceylon, South Africa, France, England, Scotland, Wales, Canada, USA, Malta and South America. One of the largest of these systems was installed in Longchamps Paris in 1928 with 273 terminals. This was a real time, large scale, multi terminal, multi user, system in 1928 long before the invention of the world’s first electronic computer. Other large installations were in Caracas and White City which ended up with 320 terminals. By 1970 after Julius tote manufacture had ceased, replaced by computer totes, Automatic Totalisators had systems operating in 29 countries.
The first Julius tote operating at Eagle Farm was installed in 1917 for the Queensland Turf Club. This was the third automatic totalisator to be installed in the world. This system, was a descendent of the World’s First Automatic Totalisator which was installed for the Auckland Racing Club at Ellerslie in 1913. The world’s second was installed for the West Australian Trotting Association in Perth. Circa 1947/1948, according to a photographer who photographed the installation in progress, a more modern descendent of the original invention was installed at Eagle Farm for the Queensland Turf Club as well as Doomben for the Brisbane Amateur Turf Club. Another Julius Tote was installed at Bundamba in 1950 for the Ipswich Amateur Turf Club. The second Julius tote installed at Eagle Farm is the one on display in the Eagle Farm Racing Museum.
The Julius Tote inside the Eagle Farm Racing Museum. Photograph by Prof Norman Heckenberg
On the subject of how long these systems dominated the totalisator business, the Julius totes had life spans far in excess of the computer totes. It was a long held belief that the last Julius tote to cease operation was at Harringay, a London dog track, in 1987. In 2005 I received an email from an engineer in Caracas. He wrote that his company had been appointed by the Venezuelan Racing Authority to investigate the workings of the Julius tote there to determine what adjustments can be made to bring it up to modern day standards. At that time the Caracas Julius Tote system had been working for 48 years!
After the Julius totes were superseded by the new computer tote systems in Brisbane, I saw them demolished over the ensuing years as the tote houses in which they resided were demolished. The bulldozers would run over the rubble of these buildings with the Julius totes below. This was the fate of all but one which is the one in the Eagle Farm Racing Museum. This museum is housed in the old Julius Tote machine room, with the tote as the centerpiece of the exhibits. The Julius Tote on display operated in its present location since circa 1947/1948 till it was superseded in 1979. You can read more about the Julius Tote in the museum at http://members.ozemail.com.au/~bconlon in the Eagle Farm Racecourse Museum chapter.
As the Eagle Farm Racing Museum is housed in the Julius Tote machine room in which the Julius Tote exhibit operated, it is interesting to have a look at some recollections of Neville Mitchell, a long serving Automatic Totalisators Manager, regarding life in these machine rooms. I find it an interesting contrast to working conditions of today. Neville said “The strictness with which the engineers ran these systems was somewhat akin to a military operation, they really had a lot of power. They had a lot of routines set down and to be an apprentice in those days was a lot of sweeping the floors and making the tea for a long long time before you actually got your hands on any piece of equipment. And I believe in the early days in Melbourne, if an apprentice was seen with his hands out of his pockets in the machine room, he would get a swift slap around the ears. The same thing applied in New Zealand. I read some stories from there and I actually knew a couple of the engineers and they applied the same very very strict mode of operation on their set-ups. They were extremely proud of these machines and some of them spent all of their, what you would call, idle time in routine maintenance and polishing of brass and things like that, that made these machines absolute showrooms.” The implication of the slap around the ears if caught with their hands out of their pockets was that they were not allowed to touch anything until they had learnt a considerable amount.
No look at the on-course totalisator in Brisbane would be complete without a mention of the Hodsdon tote. When I first started to work in Brisbane for Automatic Totalisators, I met Len Little. I was told that he had been operating a manual Trifecta tote for some years. I recall being shown markings on the walls where the marble channels of the Hodsdon Tote used to be at the Eagle Farm and Doomben old main tote houses. This was the only remaining evidence of the Hodsdon tote that had operated at these locations apart from the iron covers on part of the tote floor that covered the now unused, below ground level, counting rooms. Len Little writes “The Hodsdon Totalisator then operated for many years, on Eagle Farm, Doomben, Albion Park and Bundamba racecourses. Its operation was based on the gravitation of steel marbles from above the selling windows to a well at the end of the totalisator building. At this “well” position the marbles were mechanically counted, both for horse number and total. In the same operation resulting figures were progressively displayed for public view. Just prior to the start of World War 2, in 1939, Automatic Totalisators Ltd’s electric machine replaced Hodsdon’s at Albion Park. As the war developed racing ceased at Eagle Farm, Doomben and Bundamba because these courses were occupied by the Australian and American armies. Until after the end of the war, in 1945, all Brisbane race meetings were held at Albion Park and, for some time, on only three Saturdays out of four.”
I find one aspect of the Hodsdon tote particularly fascinating. Something that was written about it suggests that all real time systems, no matter how complex or simple, have exception conditions that generate dread in the people responsible for their performance. Digital computer systems inherently have exception conditions which are detected such as, division by zero which is mathematically not possible and can only be defined as a limit of infinity, and hardware error conditions. This detection directs execution to condition handling software which sometimes cannot recover from the condition and ends up informing the person responsible for the system that they have a problem. The worst case scenario is a system crash. In a real time application such as the tote this will mean that the responsible person’s day has taken a sudden and major turn for the worse. The exception condition for the Marble Tote was the “Bank Up”. The following is an extract from an anonymous document possibly written by Len. “On odd occasions a match or debris would find its way into the marble tracks resulting in a blockage. The cry “Bank Up” would go up, alerting special “marble boys” who armed with torches would locate the problem, clear the debris and try to ease the passing of the heavy balls. If they lost control as happened many times several hundred ball bearings would gather speed and crash into the cellar counting room and scatter everywhere, causing many educated guesses and head scratching as to what the total should have been”.
An additional comment on the Hodsdon tote is also interesting. “It has been said that the overall noise level in the tote caused by thousands of balls on the move was something to behold.” On the subject of sound, this is an interesting aspect of the Julius totes. One major difference between maintaining these electromechanical Julius tote systems and computer systems is sound. Computer totes as with computers in general are relatively quiet if not silent, with the sound of fans often being the predominant noise. The Julius totes were not quiet. They emanated sound from many sources from the whining of drive motors to the rumbling of drive shafts to the clicking of solenoids and relays, the clattering of escapement mechanisms and the humming of counters. It was said of the engineers who worked on these systems that they were able to determine the state of the system, particular ailments it had, the length of the queues and the proximity of the next race, just by listening to it. Despite the relative quiet of the computer systems, there was one analogy to this with the PDP11 computer totes. When the operating system crashed, it produced a crash dump summary printout on the system console terminal, logging information such as the contents of General Purpose Registers, The Processor Status Word, the Program Counter the Stack Pointer, Memory Management Registers, Peripheral Controller Registers and contents of the stack. The hard copy printout that this produced on the Teletype terminal made a staccato sound printing the short lines with many line feeds. This sound was quite unmistakable, so if you heard it, you did not have to read any part of the log to know the system had crashed. It was uncanny how many synchronicity events occurred with this computer tote system. If any comments were made about the system, we would “knock on wood”, which seemed to belay any repercussion from the system. A typical example is making a comment about how well the system had been performing recently. I recall one day when one of the RDC operators made such a comment. I could see from the look on her face the instant she completed the sentence, that she realised she should not have been so bold. Before she could wrap her knuckles on the wooden table top and utter the words “knock on wood” I heard the dreaded staccato printout heralding a system crash.
Finally, the PDP11 systems were superseded by a new ATL product called Atlas based on the Digital Equipment Corporation’s new VAX minicomputers in 1987. Automatic Totalisators Ltd had a name change to the acronym ATL. Atlas was replaced by a system called OCTS (Open Computer Totalisator System) by TAB Limited in 2003. During the VAX era, the Queensland Branch performed all the maintenance and enhancement programming for this system, not having to rely on head office for this function. The OCTS system was particularly interesting, seeing how differently a computer tote can be designed. The ATL systems were designed from the ground up, the company having built systems before a well established computer industry had developed or in even earlier years, before an electronics industry existed. OCTS was developed by a company called Race Course Totalisators and this system made heavy use of existing off the shelf hardware and existing software products. This made absolute sense and had many benefits for a company coming into the industry when computer systems and their associated electronics and software were well established.