Refined photon question, posted to Stack Exchange, let’s see if it gets crushed or discarded

Posted on Stack Exchange:

Mark Andrew Smith’s PhD thesis from 1994 examines relativistic cellular automata models. Also a 1999 paper by Ostoma and Trushyck examines this topic. One topic not discussed is the information required in a cell to represent photons in transit. Suppose we have cells arrayed in a cube so that each cell has 26 neighbors. Suppose there are N cells in the simulation. So it requires \log{N} bits to represent a cell location. If a photon in motion is currently in a cell, it’s direction can be represented by the location of the farthest cell it will reach on it’s straight-line trajectory. Any cell can originate a photon and can receive photons passing through from any other cell. So each cell must be able to represent N \log{N}  bits of information, to represent all photons in transit from all possible sources.

Question: Is there any schema that could represent the set of all photons passing through a cell using less information, with reasonable fidelity?

Question: According to the Pauli Exclusion Principle, any number of photons can occupy a single point in space. In the limit (real physical space), does each point in space contain an infinite number of photons? This would require infinite bits to represent. Storage of infinite bits requires infinite energy.   If so, does this pose a challenge to the idea, expressed in Fredkin’s Digital Philosophy, that the universe is in fact a cellular automata, with the limiting speed of light simply coinciding with the “clock speed” of the automata, i.e. the rate at which photons can move from one cell to the next?


Correct reproduction of BDM

Someone attempted to reproduce BDM, had problems and posted on CodeReview StackExchange asking for insight.  The dummies there criticized the white space and variable names in his code.  I found someone’s blog post with a correct answer and posted it.  Sanctimonious and clueless lifers on the site deleted the information.  The rules of StackExchange pretty much guarantee that narrow-minded lifers, similar to Wikipedia edit patrollers, will defend StackExchange against any useful content.  Oh well.  Here’s my answer:

OP is trying to write a Python program to reproduce a claimed calculation result of Bueno De Mesquita (BDM). There is another attempt to reproduce this calculation, in Python, by David Masad, “Replicating a replication of BDM“. Masad provided Python code, and also showed an approximately 20% divergence in the median score, starting from the same example and same inputs and same references. Jeremy McKibben-Sanders then replicated the model, with results matching BDM. Masad added a new post to discuss the coding issues which led him awry. Reading those posts and their code and comparing with above code will lead to correct diagnosis for above code.


Little League baseball: Play to win or play for development

From Kindgarten through 2nd grade, my son played in Little League baseball and all players were rotated through all positions during the game, and they discouraged keeping score.   Pitching was by machine. Come 3rd grade, things change:  The emphasis is now on winning. Players are selected for particular positions that they keep throughout the season.  One or two players are selected to pitch, and no others are trained in pitching.  The coach is a former minor league player with a focus on winning. If a kid can’t bat (unless it’s the lone girl on the team), he will signal them to walk or bunt.  My son hated it, and we just dropped out.

There are two pressures on the coach:  One is parents who want to see their kid’s team win at all costs, whose kids are docile enough to accept any position.  The other (apparently a great minority) are parents like myself, who want to see their 9-year-olds having fun and learning to play all the positions in the game.

I’ve never thought about this before, but apparently it’s a matter of some debate:

Cellular automata in physics and information quantity of a cell


I was taking a look at the 1994 PhD thesis of Mark Andrew Smith on Cellular Automata Methods in Mathematical Physics.  I could only find one subsequent paper by Smith, on polymer simulation in 1999 with B. Ostrovsky.  I assume he is no longer active.  The only other work I found was some apparently self-published work by Canadian engineers in 1999, Tom Ostoma and Mike Trushyk.  Like Smith they didn’t publish anything after 1999.  It doesn’t seem to be an actively pursued field.   The only reason I could find for this lack of pursuit was a comment on the Math Stack Exchange website by Willie Wong stating that

One of the reasons that it may be difficult to model Minkowski space based on cellular automata is that there are no “non-trivial” finite sub-groups of O(3,1), where non-trivial means that it doesn’t just reduce to just a finite sub group of O(3) via conjugation. So while cellular automata can be manifestly be homogeneous and isotropic (so admits a discrete O(3) symmetry), it becomes conceptually difficult to imagine some cellular automata capturing Lorentz symmetry.

Spring reading: The Sparrow (*spoilers*)


Mary Doria Russell’s The Sparrow, like Michel Faber’s The Book of Strange New Things, is a novel which attempts to give a realistic vision of first contact with a self-aware, intelligent alien species.  This is also a novel by an author new to writing science fiction. In Faber’s case, because he comes from what is called the “literary community” (no Wikipedia definition available), and in Russell’s case, this being a first novel by an anthropologist who had only written academic articles previously.

Russell’s aliens are, in a way, much less alien than Faber’s aliens.  Both have arms and legs, are bipedal, have a language, live in houses, and have some technology.  Faber’s aliens are otherwise as different from people as squid and hamsters.  Russell’s aliens are modelled after kangaroos and tigers.

In both cases, Russell and Faber downplay and lower the drama and strangeness of first contact.  They both use the colonial analogy, like Marco Polo first meeting the Chinese or Spaniards first encountering Aztecs.  In this analogy, the presence of others is not so astonishing and is not the focus of awareness.  Rather simply the inability to recognize social cues and differences in status hierarchy.  For example imagine Meriwether Lewis running into the Kim Jong Un after walking over a hill.  The social expectations and assumptions will be quite different, and one party may behave with a level of haughtiness and indifference, despite the novelty and strangeness of the encounter, that catches the other party quite unaware.

That, in essence, is the plot of The Sparrow.  Russell’s aliens are just not as alien as Faber’s because she makes a lot of assumptions that are Earth-normative:  The atmostphere and gravity of the alien planet are not discussed and one assumes identical to Earth’s.  There are two similar species.  One turns out to be domesticated herbivore prey.  The other turn out to be carnivore predators, who look similar as a result of evolutionarily adaptive aggressive mimicry.  The herbivores are like big cuddly kittens and have very dextrous hands and are very social and warm.  The carnivores have larger teeth and three-fingered, sharp claws, and are very hierarchical and cold.  They have different kinds of intelligence, but both species are intelligent and capable of change.

The novel adopts challenge to religious faith as a theme but somewhat tiresomely overplays it.   Both the humans being social at rest in the exploration group and the humands giving each other a hard time in the Jesuit context are somewhat heavily and stereotypically written.   The construction of the dual species, and the ecological imbalance accidentally introduced by the visiting human party are cleverly designed, as one would expect of a good scientist exploring a scenario in their domain.

I bought the sequel, Children of God, which will arrive in a few weeks.  The New York Times didn’t like it.  Russell plays out the scenario a little more with a return visit.  I’m looking forward to it!  My primary takeaways from this book:

  • First contact with aliens could play out just like first contact in the human context, for example when Christian missionaries came to Japan in the 1500’s.
  • We have to be very careful about unintended ecological impact of human ideas on alien society.  The predator/prey society depicted in Russell’s book had strict population controls and no risk of prey insurrection.   The human concept of gardening interfered with population controls and the human concept of strength in numbers and retaliation upset the political stability of the dual society.
  • The aliens might not like us, find us that interesting, and may look down on us, even if they are technologically inferior, so we have to be very careful about making assumptions about social hierarchies, status, and level of empathy.  (The essence of diplomacy, I suppose.
  • Maybe we can make asteroids into self-fueling space ships.  (Her one cool and relatively unexplored technical idea.)

Finally, note the novel is written in 1998 and she has a relatively uneven scorecard as a futurologist.  She got a few things right, like tablet computing, but mostly her timeline is way too ambitious, considering it’s 2017 as I write the following conditions were supposed to hold in 2016:

  • Students do not yet become indentured servants to pay for college scholarships.
  • Japan is not the dominant economic, military and political power.
  • Asteroids are not yet so thoroughly routinely mined that you can go to a broker for a used one equipped with engines that has just the right shape.
  • Jesuits don’t commission space explorations.

Collaboration at NCTC and in prediction markets and forecasting tournaments

I am reading Bridget Nolan’s 2013 UPenn thesis on workplace collaboration at the NCTC, and thinking about how that compares to workflow and collaboration in play-money prediction markets such as Almanis and Hypermind and AlphaCast, real-money markets such as PredictIt, and accuracy-score forecasting tournaments such as GJOpen.

The workflow at NCTC is described as follows:

  • Tasking. Analysts receive Taskings
  • Research. Analysts write, as quickly as possible, a, usually short, analysis related to the tasking. This could be either a forecast or an interpretation of a past event, but most likely a forecast.
  • Coordination. Analyst coordinates by sharing the analysis with all other analysts with a stake in the topic.  Analysts are indexed by region and functional area and home agency, so multiple analysts could have a stake in a topic.
    • Analysts must converge on a commonly acceptable text with coordinating analysts.
    • Analysts can game the coordination phase by
      • Limiting the review period to short or 0 (“Flash”) time periods
      • By inventing an exclusive “compartment” that coordinating analysts don’t have access to, and stealing the analysis by placing it in the compartment
  • Review. Once coordination converges, the analysis must then be approved and re-edited by all the hierarchical superiors of each participating analyst.  The claim is that there are 14 layers of management, which seems unlikely, but you never know. The Government pay grades go from GS-1 (the lowest) to GS-15, so maybe analysts come in at GS-1 and they have people at every pay grade.  GS-1 is $18,343 per year however, which implies that there are a lot of analysts subsisting well below the poverty level for the DC Metro Area, which also seems unlikely, but, again, you never know.
  • Publishing.  The analysis is delivered to the original client of the tasking.
  • Compensation.
    • Client Feedback. Some analysts are notified if the client likes or reads the product.
    • Performance Review.  Analysts are compensated by the number of pieces they are involved in that get published.
    • Work Time Away.  Time spent on foreign field visits, training and interagency meetings is not considered for performance and is effectively a form of compensation for writing analysis pieces.

For each published piece, this process can take from 10 minutes to several years, and it is a major source of stress and dissatisfaction for analysts.

Now let’s consider the case of prediction markets and forecasting tournaments.  First of all, there are two major areas of tradecraft which are out of control of the analyst, but which determine the quality of the overall process:

  • Question formation.  How to formulate a “tasking” which has an unambiguous answer, has a reasonable forward time period, covers all the possible outcomes, and is not overly specific to the extent that the actual answer to the question becomes disconnected from the tasking client’s original intent.
  • Question resolution.  How to decide when the conditions of the question have been met, and score the question accurately with unimpeachable sources, so that all analysts agree that the question has been closed and scored fairly and correctly.

Let’s assume that all the markets I mentioned up front are equally competent at question formation and resolution.  Then what distinguishes them are mainly

  • Scoring and compensation model.
    • GJOpen uses Brier Score.  Compensation is reputational: you can say you won a challenge.
    • Almanis and Hypermind use play money and the site pays cash to analysts.  This is called “creating a market for expertise”.
    • AlphaCast uses play money but also reports Brier Score.  It is a demo site for aficionados, some of whom have accumulated extremely large play money scores in a fairly small crowd.   GJOpen uses AlphaCast software underneath, so the pitch here is just to sell the software as an OEM to other vendors.
    • PredictIt uses real cash, supplied by the analysts, in a zero sum market.  Each question is a futures contract.  The site takes an 18% rake.
    • All sites have leaderboards for bins of related questions and overall cash or quality of forecasts.  Almanis has leaderboards for commenting and question posing (analysts can self-task).
  • Effect of compensation on forecasting style.  All sites penalize analysts for participating in questions they aren’t good at.  Accuracy sites however all the analyst to forecast in as many questions as they want.  Cash sites limit the analyst to forecasting in questions they have a remaining cash balance for.  Analysts can also lose all credibility by putting all of their cash on a single question with the wrong forecast.  It is impossible to lose all credibility in an accuracy-based site. Nevertheless, other analysts can see your general credibility by looking at your overall accuracy score.  (But of more relevance is looking at credibility by topic.) Play money compensation sites with thin participation can quickly become dominated by a few obsessive players accumulating very large balances.
  • Socialness.  The amount to which analysts on a site share information is more dependent on the collaboration features provided than on the scoring model. Reddit-like tree-formatted dialogues, notifications when others have responded, ability to notify/call out particular analysts, and ability to see other analysts scores, forecasts, comments and personal profiles, all have a strong impact on how much sharing happens.
  • Teaming.  In a big site with good socialness you will find that analysts form into cliques naturally and tend to have patterns of association over time.
  • Information sources.  Most analysts in public sites just use Google and are limited to what Google Search digs up.  Not much source analysis is done.  The quality of the market is dependent on the prior expertise of the participants.  Most participants simply regurgitate the most recent news as a forecast, thus acting as a Mechnical Turk news digesting machine.
  • Tools.  Public sites do not provide any kind of question domain-specific modelling tools or advanced tools for filtering and visualizing news or publicly available data.
  • Workflow.  Questions get published.  Analysts make predictions.  Questions get closed.

OK, why am I lining up NCTC workflow and forecasting workflow?  Well, the question is, what if you unplugged the Task/Coordinate/Edit/Publish model and replaced it with the Question Posing/Forecasting/Scoring model, would you get a better result?   This is the question being asked by IARPA ACE, CREATE and HFC competitions.  However it’s not clear how much of a gap there is between those competitions and current workflow at NCTC and similar places.  That is, I don’t know if there is any traction or application in the work IARPA is doing that has been translated into the actual analyst workplace.  A few observations are relevant though:

  • Seemingly large crowds in public sites boil down to a much smaller number of fanatics.  Say you have 20,000 registered users.  150 of those will forecast a lot of questions.  50 of those will be consistently accurate, and it’s not clear whether that 50 are accurate just based on survivor bias.
  • Prediction markets are sometimes terrible, especially on binary election questions. Think Scottish Referendum, Brexit, Trump.  These questions were all called wrong.
  • The particularities of intelligence analysis are not reproduced in other occupations. Forecasting markets as a work paradigm are not a toy for intelligence analysis, they are a real solution. They are a not a real solution for other occupations such as banking.
  • Analysts at NCTC are selected maybe 20% on accuracy and 80% on other factors such as
    • Commitment to training to be a professional analyst
    • Willingness and ability to pass a security clearance
    • Commitment to the occupation of being a professional analyst
  • The pre-existing workplace for analysts, with its particularities, will not go away.

The last point is most important: Analysts are a static, small population.  They can’t easily leave their jobs, and their jobs are relatively stable.  They are qualified by many other factors than accuracy.  They are a pre-existing population.  There has been much talk of “Superforecasters” a/k/a unicorns in the prediction market arena.  To adopt forecasting market methodology in intelligence analysis in a pre-existing workplace, we need to think in other terms rather than the search for these unicorns.  We have to ask: Will adopting this technology and implementing a different workflow and compensation model improve collaboration in this workplace with these people?  I think it will, at least in the sense that what Nolan describes is clearly not working well, and in the sense that the prediction market model provides a much more objective standard for scoring both the analysts and the quality of question posing and resolution.

Spring Amazon Pilots: Oasis versus it’s source novel The Book of Strange New Things (*spoilers*)


I watched the pilot for Oasis, which I liked, and immediately bought the book it’s based on, called The Book of Strange New Things.

Critics are mixed on the book.  It is literary fiction?  Is it science fiction?  Is it good fictiongood literary science fiction, bad science fiction by an arriviste?  I.e. reviewers get hung up on what category it is.  By analogy, I went to a Japanese noodle place in Charlotte that was fantastic.  “These are the best Japanese noodles I’ve ever had”, I thought.  Noodles with pecan smoked pork belly.   Spicy.  Fantastic.  Unique.  Delicious.


The catch: The chef is Italian, and learned to cook in fancy restaurants in Arizona.


So are these Japanese noodles or not?  It’s kind of like that. People get irritated when category rules are not abided by.  This same thing is happening with a new Scarlett Johansson movie, based on a Japanese manga.  Shouldn’t the heroine be Japanese?


Aren’t they appropriating?  Should straight actors be allowed to play gay men?  Should gay men be allowed to play straight men?  Should women be allowed to play men?  Should men be allowed to play women?  Like appropriating is a really nasty word.  Like we don’t appropriate all the time.  Boundaries must be respected!  So boring. Of course, I got it when Keanu Reeves pretended to be a samurai


(or was it Tom Cruise?),


but I lose patience when people get huffy about comic book adaptions.  Because face it, people, manga are comic books, written in a kind of boring, Pikachu, stereotypical style.


Get over it.

So anyway, this book, it’s like that.  Take a great science fiction genre (priest visits alien culture), and let a really good literary fiction author cook one up.  It’s good.  It’s not quite like anything a died-in-the-wool science fiction author would come up with.  It’s fresh.  So there.  I liked it.

What is really great about this book is that it does some things with space exploration and aliens which are really quite novel:

  • Space travel is depicted as painful, debilitating, messy, harsh and brief, like flying from Charlotte to Baltimore on American Airlines, only worse.  The spaceship is not pretty, dramatic or interesting, the controls are dull and simple, and it is not well decorated in any way.  It’s just dull.
  • The new world has some mystery elements but it is most of all like being stuck in a truck stop in some abandoned Southwestern town in the middle of the desert with only the most basic amenities.  It’s the Jim Jarmusch of new worlds.  There are some novel elements to the climate, the ground cover, the air, the water.  It’s different. But it’s not really exciting.  It’s a habitable place, far away, a lifeline from a dying Earth.  But it’s not great.
  • The aliens are by and large simple, pleasant and somewhat primitive villagers in some peaceful Amazonian culture untouched by modern technology.  They get by just fine.  They like aspirin, which they trade for food.  They are ugly and different, and also similar in some ways.  They can pick up our language.  We can’t pick up theirs.  They are well characterized, and different.  You rarely get to see such a depth of characterization in the science fiction genre, just as you rarely get to see the addition of really delicious pecan smoked pork belly to an otherwise standard Japanese dish that, in Japanese hands, would be rendered competently but unremarkably, and leave no particular memory.  We are used to Star Wars bar-level characterization of aliens, different on the surface.  This author goes deep on the differences, yet comes up with some believably while not jumping straight to scary. They are different, but in the way some species of squid is different from a hamster. Not like us, but not so different that there is no basis for communication.  And there is something about us that they like, and they appropriate it.  This literary description of alien appropriation of human culture is fascinating and unique. Some readers found it boring.  I do not.

The book and the TV series pilot diverge quite a lot.  Really a lot.  The TV series, in it’s hour long introduction, doesn’t really get around to the aliens until the end, and it makes them scary.  It makes a big deal out of disappearances which the book doesn’t really make a central plot point, even halfway through, where I am now.  The TV show centers a lot more on the characters at the human base.  The book centers on the alien colony, the existence of which is not mentioned at all in the pilot.

I hope the pilot gets chosen for development.  I kind of hope they follow the book.  If they don’t, they have their work cut out for them, and it will be quite a different entity from the book.  I’m not against this.  The TV Series “The Expanse” is quite different tactically from the book series it is based on.   Tactically.  But it still follows the major plotline running through the books.  It will be interesting to see whether the Amazon TV Series for Oasis will follow the same strategy, or whether it will go strategically as well as tatically different.  We already know it is tactically different.  In the TV show, the wife is dying, and is not a central focus.  In the book, the wife is not at all dying, while the world may be, and she is kind of a central, parallel narrative focus, although one of the themes of the book is how the priest detaches himself from the marriage as he gets more involved with the alien culture and the people on the human base.

Final note, in the book he invents an alphabet for the aliens or native Oasans that are central to the book (not clear how the TV show will go on this, it could throw them out completely, as depicted in the book, which may be an unhappy experience for the author but typical for Hollywood).  Writing the book he used Thai characters.  The publisher then had someone design a type font not related to any current language, so as not to offend.  Faber only actually gives us a phonic correspondence for the “s”, “t” and “ch” sounds, in cases where the Oasans are trying to pronounce English words and fail.  On occasion he makes up longer words in Oasan, unpronounced and in so doing introduces another 10 or so letters that we don’t have sounds for.  I had hoped there would be more clues and I was prepared to reproduce the full alphabet here along with English phoneticisation of the untranslated Oasan words, but there just aren’t enough examples and when they are trying to speak English, they only stumbled on “s”, “t” and “ch”.  So that’s disappointing. But apparently this was not the most fun part of writing the book for him, he didn’t really care about it that much, it was just a gimmick to convey a little more otherness.  He also gives some rules for the Oasan grammar, in particular:

  • A few thousand words
  • No cases
  • No distinction between singular and plural
  • No genders
  • Three tenses: Gone, here, and expected to come