Tweaking ACH for FARC

bokeh_plot

[Cross post]

ACH is a methodology which scores Inconsistency but doesn’t score Consistency. At most it says that evidence is really not consistent with the outcome, without opining on whether it is consistent.https://en.wikipedia.org/wiki/Analysis_of_competing_hypotheses http://competinghypotheses.org/docs/ACH,_Step_By_Step has Credibility and Relevance scores

The Xerox ACH implementation http://www.pherson.org/PDFFiles/ACHTechnicalDescription.pdf
has Credibility and Relevance scores of Low=1/sqrt(2), Medium = 1 and High = sqrt(2). The Consistency score is Very Inconsistent = -2, Inconsistent=-1, and Neutral, Consistent and Very Consistent are 0. The Weighted Inconsistency Score is Credbility * Relevance * Consistency. So any evidence item which is not inconsistent gets a weight of 0.

For my ACH-ish model, as applied to rationales with comments supplied for forecasts, I will weight as follows. Let the forecast be F. Assign Credibility and Relevance scores of Low=1/(2sqrt(2)), Medium = 1/sqrt(2), and High = 1. Let the Consistency score C be -1,-1/2,0,1/2,1. Let W = Credibility * Relevance. Then my ACH-ish-model weighted forecast will be W100*(C+1)/2 and ignore the original forecast in the formula, just looking at it as a shorthand for the text and how to view the import of the text.

So far I see 74 forecasts for FARC of which 32 have rationales. My model for FARC is ACH-ish as above. The Consensus has been towards 0 on FARC except it just picked up a little. The early comments on FARC anticipated that pick-up.

Advertisements

Does listening to old House music mean I am a successful executive with a substance abuse problem?

Just asking, because my habit of listening to Ce Ce Peniston, Robin S and Mary J. Blige at work is causing me to get ads on Pandora for a rehab facility catering to high-flying C-suite types hiding severe addictions.

My solution plan for the GF Challenge

I am a solver in the IARPA GF Challenge.  This is a cross post of my solution plan.  I am sharing mostly because I am afraid of having no competition at all.  About 500 have signed on but they are all notably silent.  Usually an indication that people are getting ready is that they try things and ask questions.   Only a few people are posting or asking questions in the GF Forum.   I don’t take this as a sign that everybody else is busy.  I take it as a sign that nobody has a clue.  We’ll see when things get started, how many people pop up on the Leaderboard.  Anyhoo, here’s my plan:

The rules say: “Finalists who wish to compete for monetary prizes will need to provide the solution package for review [by] the GF Challenge Team” and “Solvers will provide a short (4 page) explanation for their solution”.

There are several sources of information for each question:

  • Individual: Individual crowd forecasts
  • Consensus: Crowd consensus forecast
  • Model: Output of domain-specific models for the question
  • Intel: History of reference data required to resolve the question, and events and opinions gathered from news and social media
  • Crowd Experience: Brier scores of crowd forecasters identified by their anonymous GUID and recorded forecasts on resolved questions. This information becomes available as we work through and resolve the 175 questions to be answered during the Challenge.
  • My Experience: Applying my own domain-specific knowledge to the question, putting myself in the role of Analyst Manager.

I will create the follow kinds of domain-specific models, which take Intel as input (not Indiviudal or Consensus forecasts or derived Crowd Experience information):

  • Time series rate: Will Gold price lie between $1250 and $1350 on 18 March 2018?
  • Time series frequency: Will there be a mass killing event in Sudan in the month of March 2018?
  • Selectorate: Will Scotland vote to secede from the UK on or before 18 March 2018
  • Multiple Criteria Decision Analysis (MCDA): Will either Turkey or Russia officially suspend or cancel the Akkuyu nuclear power plant project before 18 March 2018?
  • Analysis of Competing Hypotheses (ACH): Will President Putin meet with Prime Minister Abe in Japan before 18 March 2018?
  • Social Media (SM): Will the UK vote to exit the European Union before 18 March 2018?

I distinguish MCDA from ACH largely by whether the outcome is predictable by reading the news or whether there is something more structural and quantifiable about the decision factors. I reserve MCDA for questions with structure and ACH for news-driven questions. Typically a power plant decision would have some structural features that can be quantified. Whether or not Putin goes to Japan is more of a guessing game signalled by diplomatic moves reported in the press. I reserve the Social Media category for matters so unpredictable that only keeping an ear to the ground (munching Twitter posts) will give an idea of the sentiment of the selectorate.

On a question by question basis I may use more than one model, and combine their results with a linear weighted average.

I will use the following prediction methods which combine the above sources of information in various ways to get a forecast stream which yields a better Brier score than the Consensus forecast stream:

  • crcc01: Consensus 0/1. Round the Consensus forecast to 0%/100%
  • crcdmf: Expert crowd (see https://larswericson.wordpress.com/2015/12/22/double-median-filter/). Take all the forecasters in a question who have beat the crowd in some related questions. Each forecaster’s weight is the sum of the accuracy scores in the questions they beat, so the more questions they were in, the higher their weight. Take the 24 forecasters with the most negative sum of accuracy scores. Take forecasts which are within 2 standard deviations of the median. Take the median again and take remaining forecasts with 2 standard deviations of that. Then take the average of those forecasts, and thats the prediction.
  • crcdmf01: Expert crowd 0/1. Take the output of double median filter and round it to 0%/100%.
  • crcm: Models. Take the output of domain-specific models only and ignore the crowd and consensus forecast stream.
  • crcm01: Models 0/1. Take the output of domain-specific models and round it to 0%/100%.
  • crcmcm: Managed consensus plus models. Combine the Consensus forecast and models using weighting decided by analyst manager (me).
  • crcmcm01: Managed consensus plus models 0/1. Round the managed consensus plus models to 0%/100%.
  • crcmem: Managed expert plus models. Combine the Expert crowd output with models using weighting decided by analyst manager (me).
  • crcmem01: Managed expert plus models. Round the managed expert plus models to 0%/100%.

Crispy salty egg

IMAG1959

There is nothing like an egg fried crispy with salt. One matter I have pondered over time is how to get the egg to spread out flat in the pan. My inspiration: Decant it from the shell from a foot or two above the pan, rather than up close. Drawbacks and considerations: Don’t put too much oil in the pan before attempting this method.

IMAG1961.jpg

Rashomon, North Carolina style

bedroom-monsters

The first report came in:

I hear there was a home invasion on the next street over and 3 suspects at large. Can anybody add information? Police were in the neighborhood investigating.

and then:

The police stopped on Kimberly and asked the kids to get inside and spoke to one of the moms. Same info. Home invasion suspect at large.

and then:

This is very concerning. Were the suspects armed? Description?

and then:

About 10 years ago, there was an attempted kidnapping of a teenage girl from the driveway of the first home on North Downing in McConnell. She was outside watching her cat about dusk when a man approached her and started a conversation. He then grabbed her and tried to run carrying her. She fought and escaped.  I don’t think he was ever caught. It was very bizarre, but look what can happen right next door to our homes.

and then:

According to our neighborhood’s Facebook page, this was a case of 3 Hispanic yard workers being mistaken for bad guys. Please let’s be careful to correctly understand situations. The workers could have been put at great risk through misunderstanding.

and then:

Long story short–a concerned neighbor called the police to my home last evening after some unusual circumstances. However, we do NOT believe there was an actual B&E that occurred. I believe my concerned neighbor was putting my mail on my front porch when he observed my back doors open and heard an unfamiliar male voice yelling to my barking dogs. After talking to some other neighbors, the unfamiliar voice is believed to be a neighborhood child who was cutting through my back yard to the front yard to catch up with some friends who were already on the street. The neighbor that called the police did not know I was up the street with my children playing in another yard. As I am reading the previous comments, I believe any non-residents within the vicinity were quickly being investigated. For instance, within 30 seconds of 3 police vehicles coming down the next street, one police vehicle made an abrupt U-Turn and sped back out on our street. In hindsight, I believe within seconds there were two calls coming across the radio, one from my house and the other from a different report of people in the vicinity. I don’t think they are related. Unfortunately, sometimes, word travels quickly and lots of information is exchanged before the interviews and details are conveyed. The officers were wonderful. I am sure they were thinking the worst when they drove by a big group of kids, playing happily before dinner, all while not knowing if there was danger present. Thankfully, there wasn’t. But, those officers and detectives entered my house to sweep it without a hesitation–putting their lives on the line.

and then:

I think we are all a little jumpy these days! One night, around 10 pm, I saw the silhouette of a man walking around outside our townhome on Caldwell Lane Had the phone in my hand to call 911! He saw me in the window and waved It was my husband taking the dog out!

We live in a very nice, very nervous neighborhood.  Locked doors and alarm systems prevail.  I don’t think I ever saw a locked door when I was my children’s age (7 and 10), but it’s been half a century since then.

A fragment of the FORMEL Library

In 1988 I was a guest of the FORMEL Project at INRIA, as a wandering ABD.  The forte of FORMEL was the unification algorithm, and I spent some time copying papers from a filing cabinet which, after my time, grew to be 3 or 4 filing cabinets and became known as the Formel Library.  I hear that, as the project disbanded, the cabinets were donated to another institution and then subsequently lost.  I copied 56 papers from the library and put them into 5 themed binders.  Recently I was possessed of a desire to clean out a storage space I have been renting, to save money by giving back the space.  The storage space is filled with items I can’t part with and never consult, including those 5 binders.

The obvious thing to do would be to scan and upload all of the papers to something like ResearchGate.  However, this thought was nixed by the Academia StackExchange community.  But what if there was something rare, valuable and irreplaceable in my collection?  Well, I inventoried what I had, and, alas, there is no such animal.  Almost all of the papers of any merit are available either freely or for large sums of money on the Internet.  What if everything academic should be free?  A workaround occurred to me: Summarize the essentials of rare or paywalled papers.  I got an OK on this idea from the community, but I don’t have the time or the desire to carry it out.  I also can’t guess how the library evolved after I left.

The intellectual property of my sample is not lost, even if the ideas have, to some extent, lost their hold on the world: One of the chief products of Formel was the CAML language, which, alas, has been pushed out on the rankings by it’s closest rival, Haskell, which is, itself, at the bottom of the list.

My binders were organized into 5 topics:

  • Unification on first-order terms
  • Derivations of the unification algorithm
  • Knuth-Bendix critical-pair completion algorithm
  • Completion in equational systems and term rewriting systems
  • Complexity and expressibility of unification and completion

Here is what they contain (or contained, if I end up recycling them):

Unification on first-order terms

Derivations of the unification algorithm

Knuth-Bendix critical-pair completion algorithm

Completion in equational systems and term rewriting systems

  • Automatic proofs by induction in theories without constructors. 1989. Information and Computation. Jouannaud, Jean-Pierre. Kounalis, Emmanuel. https://www.sciencedirect.com/science/article/pii/089054018990062X
  • A general completion algorithm for equational term rewriting systems and its proof of correctness. 1983. CRIN. Kirchner, H. No PDF found online.
  • Reveur-3: the implementation of a general completion procedure parameterized by built-in theories and strategies. 1987. Science of Computer Programming. Kirchner, Helene. Kirchner, Claude. https://www.sciencedirect.com/science/article/pii/0167642387900049
  • An Overview of Completion Algorithms. 1985. EUROCAL ’85. Llopis de Trias, Regina. No PDF found online.
  • Canonical forms for residue classes of polynomial ideals and term rewriting systems. 1983. Submitted to JACM. Llopis de Trias, Regina. No PDF found online.
  • Complete Sets of Reductions for Some Equational Theories. 1981. JACM. Peterson, Gerald E.. Stickel, Mark E. PDF behind paywall.
  • On word problems in equational theories. 1987. INRIA. Rusinowitch, Michael. Hsiang, Jieh. https://hal.archives-ouvertes.fr/inria-00075875/document
  • Equations and rewrite rules. 1980. Stanford U CSD. Huet, Gerard. Oppen, Derek C.. http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.397.599&rep=rep1&type=pdf
  • Completion of a Set of Rules Modulo a Set of Equations. 1986. SIAM J. Comp. Jouannaud, Jean-Pierre. Kirchner, Helene. PDF behind paywall.

Complexity and expressibility of unification and completion

Jury Duty

qtq80-notrp8-1140x760

I was seated in a criminal jury panel in Charlotte, in a case of felony gun ownership to be proved by constructive possession.  The prosecutor asked me these questions:

Do I know any police officers? (Yes: I practiced Aikido with some in New York.  No: I didn’t socialize with them off the mat.)

What kind of law does my wife practice? (Matrimonial and litigation, in New York.  No, not in North Carolina, she is not in the bar.)

Have I ever been in this courthouse? (Yes, to pay a speeding ticket.)

Who gave me the ticket, state police or CMPD? (Neither, my local town police.)

Was I treated unfairly?  (No, I was going fast.)

How was the case disposed? (I hired a good lawyer and she pled it down to Improper Equipment.)

I asked one question: “Can you please repeat the definition of constructive possession?  Does that mean in the house or lent to another person under control of this person?” (Yes, like that.)

Two ladies next to me confessed to owning guns: An AR-15 and two pistols each.  One used to be military police. The other was a local town cop.  The former MP, who was covered in tattoos and was once kept in jail over a dispute over jewelry with her mother, and another woman who had a vacation coming up in a few days, and I were recused under peremptory challenge by the prosecutor.  (The woman with vacation was denied recusal for cause by the judge and then given it by the prosecutor.)

When you look up peremptory challenge, most discussion is about recusal due to race. In this case, the prosecutor was a black woman, the defendant was a black man, and I am white.  So I don’t consider race an issue.  The MP was recused clearly because of unpleasant experience with the court system.  Why was I recused?  According to this article, probably I was too verbal.

Here are a couple of funny videos on jury duty (note: I wasn’t trying to get out of jury duty; I was just being myself):