Teaching Tips: Reflection Questions and Open Discussion

Allgood et al. (2016) note that, “The evidence on the use of classroom experiments and cooperative learning shows consistent positive effects on student achievement …”. This is an exciting conclusion since our view is that the best is still to come. We don’t truly understand the conditions under which experiments will have the greatest impact.

Last fall we reported on a study from Cartwright and Stepanova (2012) that showed the importance writing prompts. I wanted to expand on the idea of writing prompts by talking about Open Discussion. At the end of each of our modules we include a Final Reflection Survey. These questions help to set the stage for discussion by having students think through a problem before we initiate discussion. Questions often take the following form:

  1. Students are asked to compare two games
  2. Students are prompted to think about the assumptions of the game.
  3. Students are asked to generalize the insights of the game.

Let me provide a couple examples. Students are asked to compare standard Bertrand to Bertrand with communication and indefinite repetition (here). In the labor market with and without unemployment insurance students are encouraged to think about how changing the assumptions of the game would affect predictions (here). In the externalities with policy interventions students are encouraged to think about the assumption that social cost is known and the tax will be implemented at the correct level (here). If you’re playing our matching market you might consider asking what other labor markets would be a good fit for a matching algorithm. Or if you played the Ultimatum game you might ask about fairness as part of a general behavior function. Where is fairness most salient? When is fairness not a consideration?

The bottom line: Games spark curiosity. We can draw out that curiosity in students through good post game reflection questions and open discussion.

Does Innovative Pedagogy Matter in Classrooms?

The stage was set in Foellinger Auditorium, one of the most recognizable landmarks as well as the largest lecture hall on campus at the University of Illinois at Urbana-Champaign (UIUC). Hundreds of students streamed in, business as usual, finding their friends and favorite seats on either the main floor or the balcony. Unbeknownst to the 700+ students, they soon would be in a classroom experiment of labor market.

Foellinger Auditorium

The instructor was Dr. Jose Vazquez-Cognet, a popular educator on campus and Coursera and a pioneer in the application of classroom technology. For over a decade, Dr.Vazquez has taught the largest principles courses at UIUC, often engaging more than 900 students at a time. He has risen to the challenge every year, and used a multitude of methods: paper-and-pencil experiments (he brought thousands of pages of handouts to the class and wrote a paper about it!), real-time polling, and a multimedia solution, Flipit Economics, which he co-developed with another enterprising economist, Eric Chiang. This time, however, Dr. Vazquez had another plan.

Leading up to the day’s classroom activity, Dr. Vazquez had worked closely with MobLab’s economist Doug Norton on the Production, Entry and Exit game. In this game, students play the role of a rideshare driver, with each student having to first decide on whether to incur a fixed cost to work that day (i.e. whether to enter the ‘market’) and second, on how many hours to drive (there is increasing marginal cost from driving). When deciding to enter the market students should consider the level of competition.  The more competition there is, in the form of other drivers deciding to enter the market, the less marginal revenue for every driver. The game helps students gain insights on profit maximization as a firm as well as long-run competitive equilibrium.

Simply having his students participate in the MobLab Production, Entry and Exit game was not enough for Dr. Vazquez. He had an interesting twist on the game based on his class environment:

“Most of my students sit on the main floor, while a couple of hundred always like to chill on the balcony. Normally, I’d like to get them all down to the main floor so they can see the slides and hear me better. However, in this case, I can take advantage of this seating layout for a natural experiment.

I can create two ridesharing markets, one called ‘Main Floor’ and the other called ‘Balcony’, and ask my students to join the corresponding market based on their seating location. Based on the density of the student participation in each market, what should happen?

We would have two labor markets with similar conditions, except one is packed more densely with potential ‘drivers’ than the other.”

Dr. Vazquez quietly watched as students flooded Foellinger Auditorium from all directions. As expected, the main floor was crowded with students, while the balcony was only sparsely occupied. Next Dr. Vazquez explained the Production, Entry and Exit game to his students, including that students would earn extra credit proportional to their profit as a driver. He asked students to access their MobLab accounts and join the corresponding experiment session based on where they were seated, ‘Balcony’ for those in the balcony, and ‘Main Floor’ for those in the lower level. Finally, before he began the game, he asked students to predict which market would earn drivers more profit on average (Figure 1).

Figure 1. Polling results of student predictions on profit comparison between the two ridesharing labor markets.

Figure 1. Polling results of student predictions on profit comparison between the two ridesharing labor markets.

As the game started, you could literally hear a pin drop in the auditorium! We have visited hundreds of classrooms around the world, and it is always amazing to see that when we involve students in our lectures, even the most ardent YouTube and Facebook fans can spare a few minutes to learn.

UIUC Economics lectureDr. Jose Vazquez-Cognet UIUC Production Entry Exit Economics Lecture,

After a practice session, the game was repeated for another five rounds. Data from the two markets were shown in the following table.

UIUC Production Entry Exit Data

When the students saw the summary statistics on the screen, the murmurs grew louder. Dr. Vazquez then drove home the message: in light of what you see, would you like to go upstairs to the Balcony market? A natural experiment of two labor markets equilibrating unfolded right before the eyes of the students.

Technology alone cannot change education. How instructors apply it, in both traditional face-to-face and online classrooms, is just as important. In Dr. Vazquez’s class, he spent time to prepare the experiment, came up with a game plan for his classroom environment, and relentlessly focused on student engagement. By turning an almost unmanageable auditorium into a stimulating natural laboratory and leveraging technology, he left his students with a memorable learning experience.

We had a front-row seat to what teaching would look like in the future. Thank you, Dr. Vazquez!

UIUC Production Entry Exit

1.     Vazquez-Cognet, Jose J., Running Classroom Experiments in a Large Enrollment Economics Course (August 28, 2006). Available at SSRN: https://ssrn.com/abstract=927028 or http://dx.doi.org/10.2139/ssrn.927028

MobLab End of Term Survey

Learning is about feedback. Our platform, and its supporting content, become better when instructors and students provide their insights. In that spirit we want to make our instructors aware of the MobLab End of Year Survey.

You can find the MobLab End of Year Survey in the surveys tab shown below.


Once you select the surveys tab choose the surveys category to find MobLab End of Year Survey. Then, add it to your playlist.


Be sure to set the “Game Duration” to a desirable time. We recommend giving students 48 hours to complete the end of year survey. Soon you will have information on whether students find the games helpful for their learning and you will get a sense of what students think about the technology and how it was integrated into the class.

Thank you for running the MobLab End of Year Survey with your students!

Teaching Tip: Sequential Presentation of Data

You all know that in MobLab graphs are available immediately after game play. But, how much have you thought about how to present the information those graphs contain? In this blog post I argue that presenting information sequentially is a good way to talk students through theory and data. Let me show you what I mean with a quick example from the Competitive Market game.

When you click the “Results” button you see a graph with all lines present. However, you can use the legend below the graphs to add and subtract pieces of information as shown below. This is the crux of this blog post.



Let’s start with the Competitive Market Game:

  • Using the checkboxes in the legends for these two graphs, hide all of the lines.
  • Reveal the market demand curve. Ask students what it represents and ask them to provide reasons why it slopes downward.
  • Reveal the market supply curve. Ask students what it represents and ask them to provide reasons why it slopes upward.
  • Reveal the equilibrium price line. Ask students to explain why it is an equilibrium. If students simply give a textbook response probe them about the game. Often buyers start out submitting stingy bids and sellers hold out for high prices. Across time sellers must undercut each other to find willing buyers. Buyers must bid up each other to find willing sellers.
  • Is the equilibrium price-quantity pair a good prediction?
  • Reveal the transaction markers.

This sequential presentation of the data helps review content and walks students to the prediction. Then showcasing the data emphasizes economics as an empirical science.


Tragedy of the Commons at Penn State

After the recent RMU Teaching Workshop I road tripped over to Penn State with the amiable and generous Jadrian Wooten. I had a blast. We talked economics and life and he introduced me to Primanti Brothers which was delicious (the off-menu “When Pigs Fly” sandwich was my kind of blue collar food).

On the following Monday we ran three versions of the “Commons: Fishery” game which went really well. Note, I’m only presenting a subset of results as there were many students.

Version 1: N=1. The initial stock of fish is 40 and the fish stock doubles at the end of each round. The goal of this parameterization was two-fold. First, I wanted students to be familiar with the replacement rate and what that implied about sustainable catch each round. Second, I wanted a baseline for comparison when property rights are not so clean. From the results below you can see many students tracked well with the sustainable rate of resource consumption.

Screenshot 2017-03-15 13.52.51

Version 2: N=4. The initial stock of fish is 160 and the fish stock doubles at the end of each round. While students have now been familiar with the replacement rate the incentives are clear, “I drink your milkshake!“:

Screenshot 2017-03-15 13.56.33

After they participated in Version 1 and Version 2 I asked the following question on our free-text response, “Compare the second game, when you fished a common sea (with three others), to the first game when you fished alone. Explain any differences in incentives across those two games and whether those differences affected how you played.” Here are some student responses:

“The first game I fished sustainably, in the second – I tried sustainability at first but soon found I was competing to be able to get as many fish as I wanted and supply quickly ran out.”

“Property Rights, first one you have property rights over your fish so they are protected no one else is going to take them. Second time there are no property rights so you have to guarantee that if you try to conserve fish that they will not be taken by another fisher.”

“When you fish in your own private pool, you’re the only one setting the limit as to how much you want to catch. Whereas when you’re in a pool with 3 other players, there is a lack of communication in the sense that some people have no limit and just go wild. When there is no communication, strategy just becomes irrelevant, and you’re praying to god that the people you’re paired with aren’t idiots and consume the whole pool in a frenzy to get the top score.”


Version 3: Same as version 2. This time with chat and indefinite repetition to allow them to experience a situation with more realism: neighbors can talk to each other, there are long time horizons and reputation effects, etc.

Screenshot 2017-03-15 13.59.19

Asking the students about about why communication mattered, students responded that they could coordinate with each other. This kind of coordination on a set of rules for fishing matters even if communication is cheap talk. Also, students got a laugh when I asked them to recall, “What’s the nastiest thing someone said to you over chat?” As noted in the classic paper, “Covenants with and without a Sword: Self-Governance is Possible”,

“A striking aspect of the discussion rounds was how rapidly the subjects, who had not had an opportunity to establish a well defined community with strong internal norms, were able to devise their own agreements and verbal punishments for those who broke those agreements.” (Ostrom, Walker, and Gardner, 1992, p. 410).

Jadrian noted that 61% of students found MobLab added value to the lecture, 30% were neutral, and 9% didn’t like it. But, what I found really interesting was his comments after talking to his good students, “they enjoyed getting to see the results of what we talked about rather than just thinking about how it would happen.” This is an important point: MobLab isn’t just a useful tool for engaging students that are difficult to engage in standard lecture. MobLab engages good students because it showcases economics as an empirical science and not just a bunch of theories. 

I am hopeful the game made the incentive problems clearer to them but also gave them a sense that incentives are not destiny. Cheap talk can help construct rules and verbal reprisals can act as a price on the externality we impose on others. Economics is a rich science that can help them explore whatever issues they’re passionate about when they “look out the window.” Many thanks to Jadrian for entrusting his class to me.

A Personal Note on Match Day and our KR Matching Game

My wife is a pediatrician. In mid-March 2013 we experienced “match day”: the day where senior medical students match to a specialty-hospital pair for their residencies. For a moment stroll with me down memory lane: All the soon-to-be doctors were dressed up with family and friends in tow. There was so much chatter, excitement, hugs —the whole place was electric. The ballroom was decorated in a “Match Madness” theme and medical school faculty dawned basketball and referee uniforms with whistles and headbands. Each senior medical student was handed an envelope with their match location and speciality inside. And though I wanted to tear the thing open we were told not to open it until the appointed time.


When the envelopes were opened it is difficult to communicate the emotion. So much time invested in medical school and so many great and terrible experiences in the process of becoming a doctor. And, the hope after visiting so many different residencies, that you would get your preferred choice. We were very happy when we found out we were headed to Greenville, SC. The program there is stellar with committed attendings and a 100 percent pass rate on the pediatric boards for the last several years. That’s better than any pediatric residency east of the Mississippi.

So how did “match day” come about? In 1952 John Stalknaker and F. J. Mullen designed the National Residency Matching Program (NRMP) to alleviate problems in the labor market for physicians. The problem (described here) was,

“…competition for scarce medical students forced hospitals to offer residencies (internships) increasingly early, sometimes several years before graduation. Matches were made before the students could produce evidence of their qualifications, and even before they knew which branch of medicine they would like to practice. When an offer was rejected, it was often too late to make offers to other candidates …”

The NRMP required both students (hospitals) to submit a ranking of their most desired hospitals (students). Then an algorithim would match students and hospitals. In 1984, economist Al Roth showed that this algorithim was strategy-proof and produced “stable” matches. A match is stable if nobody perceives that a swap between residents and hospitals would be mutually beneficial. Later Al Roth would make modifications to the NRMP to accommodate married doctors. This is part of the work that eventually won Al Roth the 2012 Nobel Prize, “for the theory of stable allocations and the practice of market design”.


Speaking of Al Roth, he also happens to be a Senior Scientific Advisor at MobLab. In 2016, after a long day of job interviews and climbing hills in San Francisco I sat down for dinner at Le Colonial with the folks at MobLab. When he sat down at the table I was starstruck. Not knowing what to say, I leaned over and shook his hand,  “Thank you, Professor Roth. My wife and I are very happy with where she matched.” He laughed and smiled. He was kind and gracious in conversation and told me, “call me Al”. That is a dinner I won’t forget.

It pleases me greatly to announce that soon MobLab will be releasing its KR Matching Game (the KR stands for Kagel and Roth and refers to this paper in the Quarterly Journal of Economics). In this paper Kagel and Roth study the decentralized matching system that “unraveled” (with early matches and miscoordinations) as well as centralized systems like the NRMP. Their experiments provided a greater degree of confidence that what they had observed in a handful of cases were robust.

In our new game we will allow our instructors to run students through different matching markets to allow students to experience market unraveling and the different solutions to that problem. Keep a look out for the game in the “Market” tab in your instructor console.


Finally, it’s worth noting that the problem of matching is ubiquitous and far beyond NRMP. As Al Roth states in his excellent Freakonomics interview,

“Matching markets are markets where money, prices don’t do all the work. And some of the markets I’ve studied, we don’t let prices do any of the work. And I like to think of matching markets as markets where you can’t just choose what you want even if you can afford it — you also have to be chosen. So job markets are like that; getting into college is like that. Those things cost money, but money doesn’t decide who gets into Stanford. Stanford doesn’t raise the tuition until supply equals demand and just enough freshmen want to come to fill the seats. Stanford is expensive but it’s cheap enough that a lot of people would like to come to Stanford, and so Stanford has this whole other set of market institutions. Applications and admissions and you can’t just come to Stanford, you have to be admitted.”

Thinking through how different institutions or “rules of the game” affect market outcomes is really important. Behind all that math there is a human component. My wife and I are almost certainly happier with our outcome than we would have been before 1952.

Livin’ the Mug Life: Introducing the Beer Game

Ever go to the store to pick up the “beer essentials” and wondered how they got there? There are vast networks of exchange in place to get the simplest of items to our grocery store. It’s really amazing when you think of it.


Our “beer game” helps students understand how supply chains operate and to illustrate a common problem called the “bull whip effect” that comes from information asymmetries. This effect is characterized by how sudden changes in customer demand generate larger and larger changes in order submissions upstream through the supply chain.


In this game each student represents one of four roles in a supply chain: retailer, wholesaler, distributor, and producer. As depicted in the picture above, orders flow upstream between firms and shipments flow downstream between firms. Each round proceeds in the following way:

    • You receive incoming deliveries from upstream firms
    • Update inventory
    • You receive incoming orders from downstream firm
    • Send delivery to downstream firm from available inventory
    • Place order with upstream firm
      • Note: There is a time delay for orders to be filled. An order placed in the current round (week) takes one week to travel upstream between each node in the supply chain. Deliveries travel downstream and are subject to shipping delays between each node in the supply chain.


The goal of each firm is to minimize costs from holding inventory ($1/unit) and from unmet demand or backlog ($2/unit). But, minimizing cost gets harder and harder as you move upward through the supply chain since order submissions become larger as firms seek to avoid the penalty from unmet demand. Then order submissions nose dive as firms begin to feel the pain of high inventory costs. This kind of behavior is a huge drag on supply chain efficiency.

Speaking of efficiency, while most implementations of the beer game and run by hand, game board, etc.. This game has a smooth interface that can be played on any internet connected device. Plus, graphs and summary tables with relevant data are available immediately following game play. This will help drive home the bull whip effect but also to show how logistical improvements that reduce shipping delays can attenuate the bull whip effect.

Happy Playing!


Teaching Tip: Reflection Questions Before Results

Many instructors want to increase student engagement and make their class more interactive. One way to increase student interaction with our MobLab results is to have students write a reflection on the game immediately after playing and before presenting the results. For example, before presenting the results on the public goods game ask students some questions:

  • What did you contribute in the first round? What factors did you consider when choosing how much to contribute in subsequent rounds?
  • According to an economist, how do you choose how many donuts you buy? According to an economist, how would you choose how much to contribute to the water project? How does the water purification differ from a donut?

Screenshot 2017-03-10 19.34.26

Give students a couple minutes to answer these questions. Having thought through the ideas and organized their thoughts students will be more willing to jump into the fray. In addition, there is reason to believe that many of the positive benefits from classroom experiments come from reflection questions.

If you’re interested in reflection questions the economists from MobLab have dreamed up, check out our Modules section. Happy Playing!

Conversations with Educators: Andrew Kashdan


We are continuing our Conversations with Educators series. While we primarily work with college educators there are a number of high school educators who have shown enthusiasm about MobLab. One of our earliest high school adopters was Andrew Kashdan. Below is our interview with him:

Q: First, please tell everyone a little bit about yourself and how you became interested in economics.

A: Although I had graduated with a degree in Applied Economics, my interest in understanding the subject at a deeper level really began “on the job” when I worked in the financial industry in various capacities. Economics really allows you to better understand what is going on in the real world. I ended up going back to school and earned a master’s degree in economics.

Q: When did you know you wanted to become a teacher?

A: I definitely didn’t predict my career path early on. As a graduate student, I taught some courses and found the teaching aspect more to my liking than focusing on research. It was a lucky accident that I ended up teaching at an independent school where I can specialize in economics. I have the opportunity to teach introductory (AP) Micro and Macro, as well as a Post-AP seminar in political economy that I created.

Q: You are one of the first high school teachers to have adopted MobLab. How has the platform evolved since you first started using us?

A: The main thing, which is what really adds value, is the addition of more games. I remember seeing a documentary on behavioral economics that talked about how Vernon Smith ran experiments on asset market bubbles and thought, “that would be a great game to have in class.” I was excited when Moblab created a version of it, and I think it’s one of the better games because you often get a non-obvious result.

Q: How has the experience with MobLab been from your perspective? What have student reactions been?

A:If my students had their way, we’d use MobLab a lot more. It’s a good way to break up a traditional lesson that is both fun and productive.

Q: Do you have a favorite MobLab game to play with students?

A: My personal favorite is a simple one, playing a tournament of a repeated prisoner’s dilemma. As anyone knows who has studied a little bit of game theory, it allows for some interesting experiments in trying out different strategies with slight variations, and students will discover what characteristics a successful strategy will have. Another popular one is the public goods game with punishment — students relish the opportunity to punish their classmates for not cooperating.

Public Goods: Punishments and Rewards game screen

Q: What advice would you give a new instructor who wants to use games in their classroom?

A: If their situation allows them to get access to MobLab, it seems like a no-brainer to incorporate it into an economics class. It’s fun for students, and allows them to see some of the theories they are learning in a different way. It’s one thing to show an equilibrium on a graph; it’s another to see it emerge in a game when players have limited information. I recommend going through the video instructions yourself, and experimenting with test accounts, so you can give a concise overview for your students.