<?xml version="1.0" encoding="UTF-8"?>
<rss  xmlns:atom="http://www.w3.org/2005/Atom" 
      xmlns:media="http://search.yahoo.com/mrss/" 
      xmlns:content="http://purl.org/rss/1.0/modules/content/" 
      xmlns:dc="http://purl.org/dc/elements/1.1/" 
      version="2.0">
<channel>
<title>Athos Damiani</title>
<link>https://athospd.github.io/athosdamiani/blog.html</link>
<atom:link href="https://athospd.github.io/athosdamiani/blog.xml" rel="self" type="application/rss+xml"/>
<description>Notes on data science, data engineering and code — written in Quarto with R and Python.</description>
<generator>quarto-1.11.9</generator>
<lastBuildDate>Fri, 20 Nov 2026 00:00:00 GMT</lastBuildDate>
<item>
  <title>Winding a cosine around a circle: the Fourier transform in a GIF</title>
  <dc:creator>Athos Damiani</dc:creator>
  <link>https://athospd.github.io/athosdamiani/posts/2026-11-20-fourier-transform-gif/</link>
  <description><![CDATA[ 





<p>A while ago I <a href="https://www.linkedin.com/posts/athosdamiani_rstats-rstats-python-share-6961686612699168768-cPvc/">posted on LinkedIn</a> an #rstats GIF inspired by 3Blue1Brown’s <a href="https://www.youtube.com/watch?v=spUNpyF58BY">But what is the Fourier Transform? A visual introduction</a>. This post is that GIF, with the <a href="https://gist.github.com/Athospd/e554e9090564984a527bb81db1728b0c">code</a> behind it.</p>
<p><img src="https://athospd.github.io/athosdamiani/posts/2026-11-20-fourier-transform-gif/fourier.gif" class="img-fluid"></p>
<section id="the-idea" class="level2">
<h2 class="anchored" data-anchor-id="the-idea">The idea</h2>
<p>Take a signal <img src="https://latex.codecogs.com/png.latex?g(t)">, here <img src="https://latex.codecogs.com/png.latex?g(t)%20=%20%5Ccos(20t)"> for <img src="https://latex.codecogs.com/png.latex?t"> between 0 and 1, and wrap it around the origin of the complex plane. Each point of the signal becomes <img src="https://latex.codecogs.com/png.latex?g(t)%5C,e%5E%7Bi%5Comega%20t%7D">: the angle comes from <img src="https://latex.codecogs.com/png.latex?t"> and the winding frequency <img src="https://latex.codecogs.com/png.latex?%5Comega">, and the distance from the origin comes from <img src="https://latex.codecogs.com/png.latex?g(t)">.</p>
<p>Then take the centre of mass of the wrapped curve, the average of all its points:</p>
<p><img src="https://latex.codecogs.com/png.latex?%0A%5Chat%7Bg%7D(%5Comega)%20=%20%5Cfrac%7B1%7D%7BT%7D%5Cint_0%5ET%20g(t)%5C,e%5E%7Bi%5Comega%20t%7D%5C,dt%0A"></p>
<ul>
<li><strong>left panel</strong>: the wrapped curve for the current <img src="https://latex.codecogs.com/png.latex?%5Comega"> and its centre of mass (red dot).</li>
<li><strong>right panel</strong>: the real part of the centre of mass as <img src="https://latex.codecogs.com/png.latex?%5Comega"> goes from 0.4 to 40.</li>
</ul>
<p>For most winding frequencies the curve spreads evenly around the origin and the centre of mass stays near 0. When <img src="https://latex.codecogs.com/png.latex?%5Comega%20=%2020">, the frequency of the cosine, the peaks of the signal all land on the same side of the circle and the centre of mass jumps to about 0.5. That spike is the frequency showing up in the Fourier transform.</p>
</section>
<section id="the-code" class="level2">
<h2 class="anchored" data-anchor-id="the-code">The code</h2>
<p><code>f()</code> is the signal and <code>expi()</code> is <img src="https://latex.codecogs.com/png.latex?e%5E%7Bix%7D"> written as <img src="https://latex.codecogs.com/png.latex?%5Ccos%20x%20+%20i%20%5Csin%20x">. For each <code>a</code>, <code>x</code> runs from 0 to <code>a * L</code> and the signal is evaluated at <code>x / (a * L)</code>, which goes from 0 to 1. So <code>expi(x) * f(x / (a * L))</code> is <img src="https://latex.codecogs.com/png.latex?g(t)%5C,e%5E%7Bi%5Comega%20t%7D"> with <img src="https://latex.codecogs.com/png.latex?%5Comega%20=%20a%20L">.</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb1" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb1-1">f <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">function</span>(x) <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">cos</span>(x<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">*</span><span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">20</span>)</span>
<span id="cb1-2">expi <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">function</span>(x) <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">cos</span>(x) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">sin</span>(x)<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">*</span><span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span>i</span>
<span id="cb1-3">L <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">40</span></span>
<span id="cb1-4">medias_Re <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">c</span>()</span>
<span id="cb1-5">fs <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">c</span>()</span>
<span id="cb1-6">as <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">seq</span>(<span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0.01</span>, <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">l =</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">140</span>)</span></code></pre></div></div>
</div>
<p>The original gist writes the GIF with <code>animation::saveGIF()</code>, which needs ImageMagick, and pauses with <code>Sys.sleep(0.1)</code> between frames. Here <code>gifski::save_gif()</code> records each plot as a frame instead, so <code>par(mfrow = c(1,2))</code> moves inside the block to apply to the GIF’s device.</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb2" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb2-1">gifski<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">::</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">save_gif</span>({</span>
<span id="cb2-2">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">par</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">mfrow =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">c</span>(<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span>,<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">2</span>))</span>
<span id="cb2-3">  <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">for</span>(a <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">in</span> as) {</span>
<span id="cb2-4">    x <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">seq</span>(<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">0</span>, a<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">*</span><span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">40</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">l =</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">500</span>)</span>
<span id="cb2-5">    ft <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">expi</span>(x)<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">*</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">f</span>(x<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">/</span>(a<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">*</span>L))</span>
<span id="cb2-6">    media_Re <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">mean</span>(<span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">Re</span>(ft))</span>
<span id="cb2-7">    <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">plot</span>(<span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">Re</span>(ft), <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">Im</span>(ft), <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">xlim =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">c</span>(<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">-</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">1.2</span>, <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">1.2</span>), <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">ylim =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">c</span>(<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">-</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">1.2</span>, <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">1.2</span>), <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">main =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"cos(20t) - Mass Center on Complex Plane"</span>)</span>
<span id="cb2-8">    <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">points</span>(media_Re, <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">mean</span>(<span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">Im</span>(ft)), <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">col =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"red"</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">cex =</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">3</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">pch =</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">16</span> )</span>
<span id="cb2-9">    medias_Re <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">c</span>(medias_Re, media_Re)</span>
<span id="cb2-10">    fs <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">c</span>(fs, a<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">*</span>L)</span>
<span id="cb2-11">    <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">plot</span>(fs, medias_Re, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">type =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"l"</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">xlim =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">c</span>(<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">0</span>, <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">40</span>), <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">ylim =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">c</span>(<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">-</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0.11</span>, <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0.51</span>))</span>
<span id="cb2-12">    <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">points</span>(a<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">*</span>L, media_Re, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">col =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"red"</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">cex =</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">3</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">pch =</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">16</span>)</span>
<span id="cb2-13">  }</span>
<span id="cb2-14">}, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">gif_file =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"fourier.gif"</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">width =</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">600</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">height =</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">300</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">delay =</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0.1</span>)</span></code></pre></div></div>
</div>
<p>The video goes further, from this centre-of-mass picture to the Fourier transform formula itself.</p>


</section>

 ]]></description>
  <category>concepts</category>
  <category>r</category>
  <category>fourier transform</category>
  <category>animation</category>
  <guid>https://athospd.github.io/athosdamiani/posts/2026-11-20-fourier-transform-gif/</guid>
  <pubDate>Fri, 20 Nov 2026 00:00:00 GMT</pubDate>
  <media:content url="https://athospd.github.io/athosdamiani/posts/2026-11-20-fourier-transform-gif/fourier.png" medium="image" type="image/png" height="72" width="144"/>
</item>
<item>
  <title>{targetsboard}: a Shiny + React board for {targets} pipelines</title>
  <dc:creator>Athos Damiani</dc:creator>
  <link>https://athospd.github.io/athosdamiani/posts/2026-11-20-targetsboard/</link>
  <description><![CDATA[ 





<p>This weekend’s project was <a href="https://github.com/Athospd/targetsboard">{targetsboard}</a>, an alternative Shiny + React UI to monitor and (perhaps one day) control <a href="https://books.ropensci.org/targets/">{targets}</a>’ DAGs. For those familiar with {targets}, it serves as an alternative to the default <code>tar_watch()</code>.</p>
<video src="targetsboard.mp4" autoplay="" loop="" muted="" playsinline="" style="width: 100%; height: auto;" aria-label="targetsboard showing a targets DAG updating while tar_make() runs">
</video>
<p>I absolutely LOVE the {targets} package, and I mostly did this project for fun (and it was indeed fun!). However, it will take some time to make it minimally viable, so I’m publishing it with the hope of finding skilled folks to help take the idea and develop a proper version.</p>
<section id="the-idea" class="level2">
<h2 class="anchored" data-anchor-id="the-idea">The idea</h2>
<p>The main concept of this project is to propose a data orchestration tool designed with Data Scientists in mind, emphasizing reproducibility, reporting, and a robust feedback loop. It would be fantastic to have an Airflow-grade tool centered on Data Science!</p>
<p>And honestly, I believe that R has what it takes to be a leading force in this area. R has the potential to be an enterprise-ready, first-class citizen in MLOps. Packages like {targets}, {shiny}, {reactR}, {tidyverse}, {quarto}, {vetiver}, {sparklyr}, {dbplyr}, etc., are simply AMAZING!</p>
</section>
<section id="how-to-try-it" class="level2">
<h2 class="anchored" data-anchor-id="how-to-try-it">How to try it</h2>
<p>The package is at an ultra-early, experimental stage. Install the development version from GitHub:</p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb1" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb1-1">remotes<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">::</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">install_github</span>(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Athospd/targetsboard"</span>)</span></code></pre></div></div>
<p>In your normal, day-to-day targets workflow, launch the board with <code>tar_board()</code>. It runs in the background and is served at <code>localhost:9999</code>:</p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb2" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb2-1"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">library</span>(targetsboard)</span>
<span id="cb2-2"></span>
<span id="cb2-3"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># launch the app</span></span>
<span id="cb2-4">app <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">tar_board</span>()</span>
<span id="cb2-5"></span>
<span id="cb2-6"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># go to localhost:9999 where the board is being served</span></span>
<span id="cb2-7"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">browseURL</span>(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"http://localhost:9999"</span>)</span>
<span id="cb2-8"></span>
<span id="cb2-9"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># you can check it is alive, running on background</span></span>
<span id="cb2-10">app<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">is_alive</span>()</span></code></pre></div></div>
<p>It keeps track of your targets’ metadata just like <code>tar_watch()</code> would. Run your pipeline as usual and watch the DAG update:</p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb3" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb3-1"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">tar_make</span>()</span></code></pre></div></div>
<p>When you are done, shut the app down:</p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb4" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb4-1">app<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">kill</span>()</span></code></pre></div></div>
</section>
<section id="packages-to-put-under-your-radar" class="level2">
<h2 class="anchored" data-anchor-id="packages-to-put-under-your-radar">Packages to put under your radar</h2>
<ul>
<li><a href="https://books.ropensci.org/targets/">{targets}</a>: amazing data orchestration with a focus on data science.</li>
<li><a href="https://react-r.github.io/reactR/">{reactR}</a>: a powerful templating toolkit to connect R with React components (I used the <a href="https://reactflow.dev/">reactflow</a> React lib as the DAG interface).</li>
<li><a href="https://shiny.posit.co/">{shiny}</a>: now also available for Python, a rich web app framework, awesome for us mortal data scientists.</li>
</ul>
</section>
<section id="lets-discuss-it" class="level2">
<h2 class="anchored" data-anchor-id="lets-discuss-it">Let’s discuss it!</h2>
<p>What orchestration tools do you know that are already DS-friendly? What data orchestration tool do you like the most? What are your thoughts on my beloved R in the MLOps field? Please let me know.</p>
<p>Code: <a href="https://github.com/Athospd/targetsboard" class="uri">https://github.com/Athospd/targetsboard</a></p>


</section>

 ]]></description>
  <category>r</category>
  <category>targets</category>
  <category>shiny</category>
  <category>reactR</category>
  <category>mlops</category>
  <category>packages</category>
  <guid>https://athospd.github.io/athosdamiani/posts/2026-11-20-targetsboard/</guid>
  <pubDate>Fri, 20 Nov 2026 00:00:00 GMT</pubDate>
  <media:content url="https://athospd.github.io/athosdamiani/posts/2026-11-20-targetsboard/thumbnail.png" medium="image" type="image/png" height="72" width="144"/>
</item>
<item>
  <title>Gentle introduction to the learning rate parameter</title>
  <dc:creator>Athos Damiani</dc:creator>
  <link>https://athospd.github.io/athosdamiani/posts/2026-11-20-learning-rate/</link>
  <description><![CDATA[ 





<p>I found an old <a href="https://gist.github.com/Athospd/97f16d5768ef72e87d2840f524597653">gist</a> on my GitHub about the learning rate. It fits a regression line by gradient descent, and it’s oddly satisfying to watch. A bit insightful too.</p>
<p><img src="https://athospd.github.io/athosdamiani/posts/2026-11-20-learning-rate/learning_rate.gif" class="img-fluid"></p>
<p>There are three curves, one for each learning rate:</p>
<ul>
<li><strong>orange</strong>: too low (<code>lr = 0.001</code>). Each step is tiny, so after 100 steps the curve is still far from the points.</li>
<li><strong>red</strong>: too high (<code>lr = 2</code>). Each step overshoots, so the curve swings far from the points before it settles.</li>
<li><strong>blue</strong>: just right (<code>lr = 0.05</code>). The curve moves steadily toward the points and stays there.</li>
</ul>
<section id="the-learning-rate" class="level2">
<h2 class="anchored" data-anchor-id="the-learning-rate">The learning rate</h2>
<p>Gradient descent updates the model parameters a little at each step, in the direction that lowers the loss:</p>
<p><img src="https://latex.codecogs.com/png.latex?%5Ctheta_%7Bt+1%7D%20=%20%5Ctheta_t%20-%20%5Ceta%20%5C,%20%5Cnabla%20L(%5Ctheta_t)"></p>
<p>The learning rate <img src="https://latex.codecogs.com/png.latex?%5Ceta"> is the size of that step. The optimizer below is Adam, which adapts the step for each parameter, but the learning rate still sets its scale.</p>
</section>
<section id="the-code" class="level2">
<h2 class="anchored" data-anchor-id="the-code">The code</h2>
<p>The data is the <code>cars</code> dataset: stopping distance (<code>dist</code>) as a function of speed. The model is <code>dist ~ speed + speed^2</code>, written as a linear layer with two inputs.</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb1" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb1-1"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">library</span>(torch)</span>
<span id="cb1-2"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">torch_manual_seed</span>(<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span>)</span>
<span id="cb1-3"></span>
<span id="cb1-4">x <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">torch_tensor</span>(<span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">cbind</span>(cars<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span>speed, cars<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span>speed<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">^</span><span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">2</span>))</span>
<span id="cb1-5">y <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">torch_tensor</span>(cars<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span>dist)<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">unsqueeze</span>(<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">2</span>)</span>
<span id="cb1-6">mse <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">nn_mse_loss</span>()</span>
<span id="cb1-7"></span>
<span id="cb1-8">model_gen <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">function</span>(lr) {</span>
<span id="cb1-9">  lin <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">nn_linear</span>(<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">2</span>, <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span>)  <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># b0, b1 and b2</span></span>
<span id="cb1-10">  opt <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> torch<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">::</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">optim_adam</span>(lin<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span>parameters, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">lr =</span> lr)</span>
<span id="cb1-11">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">return</span>(<span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">list</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">lin =</span> lin, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">opt =</span> opt))</span>
<span id="cb1-12">}</span>
<span id="cb1-13"></span>
<span id="cb1-14">model_fit <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">function</span>(model) {</span>
<span id="cb1-15">  model<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span>opt<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">zero_grad</span>()</span>
<span id="cb1-16"></span>
<span id="cb1-17">  pred <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> model<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">lin</span>(x) <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># b0 + b1*speed + b2*speed^2</span></span>
<span id="cb1-18">  custo <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">mse</span>(y, pred)</span>
<span id="cb1-19">  custo<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">backward</span>()</span>
<span id="cb1-20">  model<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span>opt<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">step</span>()</span>
<span id="cb1-21">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">return</span>(pred)</span>
<span id="cb1-22">}</span></code></pre></div></div>
</div>
<p>Three copies of the same model, one per learning rate:</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb2" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb2-1">model1 <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">model_gen</span>(<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">2</span>)</span>
<span id="cb2-2">model2 <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">model_gen</span>(<span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0.05</span>)</span>
<span id="cb2-3">model3 <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">model_gen</span>(<span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0.001</span>)</span></code></pre></div></div>
</div>
<p>Each iteration takes one optimizer step for each model and draws the three fitted curves. The original gist plots to the screen with <code>Sys.sleep(0.1)</code> between steps. Here <code>gifski::save_gif()</code> records each plot as a frame instead.</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb3" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb3-1">gifski<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">::</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">save_gif</span>({</span>
<span id="cb3-2">  <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">for</span> (i <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">in</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">:</span><span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">100</span>) {</span>
<span id="cb3-3">    pred1 <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">model_fit</span>(model1)</span>
<span id="cb3-4">    pred2 <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">model_fit</span>(model2)</span>
<span id="cb3-5">    pred3 <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">model_fit</span>(model3)</span>
<span id="cb3-6">    <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">plot</span>(cars, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">main =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">paste</span>(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"step"</span>, i))</span>
<span id="cb3-7">    <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">lines</span>(x[, <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span>]<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">unsqueeze</span>(<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">2</span>), pred1, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">col =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"red"</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">lwd =</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">4</span>)</span>
<span id="cb3-8">    <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">lines</span>(x[, <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span>]<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">unsqueeze</span>(<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">2</span>), pred2, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">col =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"royalblue"</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">lwd =</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">4</span>)</span>
<span id="cb3-9">    <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">lines</span>(x[, <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span>]<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">unsqueeze</span>(<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">2</span>), pred3, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">col =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"orange"</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">lwd =</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">4</span>)</span>
<span id="cb3-10">  }</span>
<span id="cb3-11">}, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">gif_file =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"learning_rate.gif"</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">width =</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">640</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">height =</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">480</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">delay =</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0.1</span>)</span></code></pre></div></div>
</div>
<p>Tune your models, folks!</p>


</section>

 ]]></description>
  <category>concepts</category>
  <category>deep learning</category>
  <category>torch</category>
  <category>r</category>
  <category>optimization</category>
  <guid>https://athospd.github.io/athosdamiani/posts/2026-11-20-learning-rate/</guid>
  <pubDate>Fri, 20 Nov 2026 00:00:00 GMT</pubDate>
  <media:content url="https://athospd.github.io/athosdamiani/posts/2026-11-20-learning-rate/learning_rate.png" medium="image" type="image/png" height="108" width="144"/>
</item>
<item>
  <title>Dropping highly correlated features can throw away the signal</title>
  <dc:creator>Athos Damiani</dc:creator>
  <link>https://athospd.github.io/athosdamiani/posts/2026-11-20-feature-selection-by-correlation/</link>
  <description><![CDATA[ 





<p>I’ve filtered out one feature of every pair with <img src="https://latex.codecogs.com/png.latex?%5Ctext%7Bcor%7D(X_i,%20X_k)%20%3E%200.97"> for most of my career. The reasoning: if two features are almost the same, keeping both adds nothing.</p>
<p>A while ago I <a href="https://www.linkedin.com/posts/athosdamiani_feat-selection-by-correlation-when-lgbmn-activity-7256640466937425920-Ivvd">posted on LinkedIn</a> a simulation where that rule fails. When the signal lives in a ratio or a difference of two features, <img src="https://latex.codecogs.com/png.latex?X_1/X_2"> or <img src="https://latex.codecogs.com/png.latex?X_1%20-%20X_2">, both features matter, even with <img src="https://latex.codecogs.com/png.latex?%5Ctext%7Bcor%7D(X_1,%20X_2)%20=%200.99">. This post walks through the <a href="https://gist.github.com/Athospd/5d60c99c4ce1a5461e7f59c26669c7d0">simulation code</a>.</p>
<div class="callout callout-style-default callout-tip callout-titled">
<div class="callout-header d-flex align-content-center">
<div class="callout-icon-container">
<i class="callout-icon"></i>
</div>
<div class="callout-title-container flex-fill">
<span class="screen-reader-only">Tip</span>Spoiler
</div>
</div>
<div class="callout-body-container callout-body">
<p>Avoid using linear correlation for feature selection. Use the random feature strategy instead: compare each feature’s importance to the importance of features that are pure noise. More details down below.</p>
</div>
</div>
<section id="the-data" class="level2">
<h2 class="anchored" data-anchor-id="the-data">The data</h2>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb1" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb1-1"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">library</span>(tidymodels)</span>
<span id="cb1-2"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">library</span>(tidyverse)</span>
<span id="cb1-3"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">library</span>(bonsai)</span>
<span id="cb1-4"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">library</span>(lightgbm)</span>
<span id="cb1-5"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">library</span>(patchwork)</span>
<span id="cb1-6"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">library</span>(corrr)</span>
<span id="cb1-7"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">theme_set</span>(<span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">theme_minimal</span>())</span>
<span id="cb1-8">f <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> glue<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">::</span>glue</span>
<span id="cb1-9"></span>
<span id="cb1-10">add_dummy_feats <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">function</span>(df, n_new_feats) {</span>
<span id="cb1-11">  n <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">nrow</span>(df)</span>
<span id="cb1-12">  dummy_feats <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">=</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">replicate</span>(n_new_feats, <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">runif</span>(n)) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb1-13">    <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">as_tibble</span>() <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb1-14">    <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">set_names</span>(<span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">f</span>(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"random{1:n_new_feats}"</span>))</span>
<span id="cb1-15">  df <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">bind_cols</span>(dummy_feats)</span>
<span id="cb1-16">}</span></code></pre></div></div>
</div>
<p>Think of <code>X_12m</code> as some quantity measured over the last 12 months, and <code>X_06m</code> as the same quantity over the last 6 months. <code>X_06m</code> is <code>X_12m</code> minus a small random amount, so the two are almost identical. The target depends only on their ratio, <code>X_06_12 = X_06m/X_12m</code>. Three pure-noise features complete the data.</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb2" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb2-1"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">set.seed</span>(<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">123</span>)</span>
<span id="cb2-2">n <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">=</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">100000</span></span>
<span id="cb2-3">data <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">tibble</span>(</span>
<span id="cb2-4">  <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">X_12m =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">runif</span>(n),</span>
<span id="cb2-5">  <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">X_06m =</span> X_12m <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">pmin</span>(X_12m, <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">rgamma</span>(n, <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0.2</span>, <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">55</span> <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">-</span> (X_12m <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span>)<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">*</span><span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">26</span>)),</span>
<span id="cb2-6">  <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">X_06_12 =</span> X_06m<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">/</span>X_12m,</span>
<span id="cb2-7">  <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">target =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">rbinom</span>(n, <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">prob =</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">/</span>(<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span> <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">exp</span>(<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">-</span><span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span> <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">*</span> (<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">-</span><span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">60</span> <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">60</span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">*</span>X_06_12)))),</span>
<span id="cb2-8">) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">mutate</span>(</span>
<span id="cb2-9">  <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">target =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">as.factor</span>(<span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">if_else</span>(target <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">==</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"a_1"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"b_0"</span>))</span>
<span id="cb2-10">) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb2-11">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">add_dummy_feats</span>(<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">3</span>)</span>
<span id="cb2-12"></span>
<span id="cb2-13">data <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb2-14">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">count</span>(target) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb2-15">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">mutate</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">p =</span> scales<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">::</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">percent</span>(n<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">/</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">sum</span>(n)))</span>
<span id="cb2-16"><span class="do" style="color: #5E5E5E;
background-color: null;
font-style: italic;">## # A tibble: 2 × 3</span></span>
<span id="cb2-17"><span class="do" style="color: #5E5E5E;
background-color: null;
font-style: italic;">##   target     n p    </span></span>
<span id="cb2-18"><span class="do" style="color: #5E5E5E;
background-color: null;
font-style: italic;">##   &lt;fct&gt;  &lt;int&gt; &lt;chr&gt;</span></span>
<span id="cb2-19"><span class="do" style="color: #5E5E5E;
background-color: null;
font-style: italic;">## 1 a_1    32522 33%  </span></span>
<span id="cb2-20"><span class="do" style="color: #5E5E5E;
background-color: null;
font-style: italic;">## 2 b_0    67478 67%</span></span></code></pre></div></div>
</div>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb3" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb3-1">cor_feats <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> data <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb3-2">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">sample_frac</span>(<span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0.1</span>) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb3-3">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">select</span>(<span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">starts_with</span>(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"X"</span>)) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb3-4">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">correlate</span>() <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb3-5">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">stretch</span>() <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb3-6">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">ggplot</span>(<span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">aes</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">x =</span> x, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">y =</span> y)) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span></span>
<span id="cb3-7">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">geom_tile</span>(<span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">aes</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">fill =</span> r), <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">show.legend =</span> <span class="cn" style="color: #8f5902;
background-color: null;
font-style: inherit;">FALSE</span>) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span></span>
<span id="cb3-8">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">geom_text</span>(<span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">aes</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">label =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">round</span>(r,<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">2</span>)), <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">size =</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">5</span>) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span></span>
<span id="cb3-9">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">scale_fill_gradient2</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">low =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"royalblue"</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">high =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"darkred"</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">mid =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"white"</span>,</span>
<span id="cb3-10">                       <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">midpoint =</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">0</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">limit =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">c</span>(<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">-</span><span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span>, <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span>)) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span></span>
<span id="cb3-11">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">coord_fixed</span>() <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span></span>
<span id="cb3-12">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">labs</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">title =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Pearson Correlation"</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">x =</span> <span class="cn" style="color: #8f5902;
background-color: null;
font-style: inherit;">NULL</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">y =</span> <span class="cn" style="color: #8f5902;
background-color: null;
font-style: inherit;">NULL</span>)</span>
<span id="cb3-13"></span>
<span id="cb3-14">X_06m_X_12m <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> data <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb3-15">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">sample_frac</span>(<span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0.1</span>) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb3-16">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">ggplot</span>(<span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">aes</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">x =</span> X_12m, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">y =</span> X_06m)) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span></span>
<span id="cb3-17">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">geom_point</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">alpha =</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0.1</span>)</span>
<span id="cb3-18"></span>
<span id="cb3-19">cor_feats <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span> X_06m_X_12m</span></code></pre></div></div>
<div class="cell-output-display">
<div>
<figure class="figure">
<p><img src="https://athospd.github.io/athosdamiani/posts/2026-11-20-feature-selection-by-correlation/index_files/figure-html/unnamed-chunk-3-1.png" class="img-fluid figure-img" width="672"></p>
</figure>
</div>
</div>
</div>
<p><code>X_06m</code> and <code>X_12m</code> are the textbook case for the correlation filter. The ratio <code>X_06_12</code>, the feature that actually drives the target, has a weak correlation with either of them.</p>
</section>
<section id="two-models" class="level2">
<h2 class="anchored" data-anchor-id="two-models">Two models</h2>
<p>In the simulation the ratio isn’t available to the model: in real life you’d have to know to build it. Both recipes drop <code>X_06_12</code>. The second one also drops <code>X_12m</code>, as the correlation filter would.</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb4" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb4-1"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">set.seed</span>(<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">123</span>)</span>
<span id="cb4-2">data_split <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">initial_split</span>(data, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">strata =</span> target)</span>
<span id="cb4-3">train_data <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">training</span>(data_split)</span>
<span id="cb4-4">test_data <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">testing</span>(data_split)</span>
<span id="cb4-5"></span>
<span id="cb4-6">lgbm_model <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">boost_tree</span>(</span>
<span id="cb4-7">  <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">mode =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"classification"</span>,</span>
<span id="cb4-8">  <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">trees =</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">100</span>,</span>
<span id="cb4-9">  <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">tree_depth =</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">5</span>,</span>
<span id="cb4-10">  <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">learn_rate =</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0.01</span></span>
<span id="cb4-11">) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb4-12">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">set_engine</span>(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"lightgbm"</span>)</span>
<span id="cb4-13"></span>
<span id="cb4-14"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># keeps both X_12m and X_06m</span></span>
<span id="cb4-15">recipe1 <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">recipe</span>(target <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">~</span> ., <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">data =</span> train_data) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb4-16">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">step_rm</span>(X_06_12) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb4-17">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">step_normalize</span>(<span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">all_predictors</span>())</span>
<span id="cb4-18"></span>
<span id="cb4-19"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># drops X_12m, as the correlation filter would</span></span>
<span id="cb4-20">recipe2 <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">recipe</span>(target <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">~</span> ., <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">data =</span> train_data) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb4-21">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">step_rm</span>(X_12m) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb4-22">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">step_rm</span>(X_06_12) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb4-23">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">step_normalize</span>(<span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">all_predictors</span>())</span>
<span id="cb4-24"></span>
<span id="cb4-25">fitted_model1 <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">workflow</span>(recipe1, lgbm_model) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">fit</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">data =</span> train_data)</span>
<span id="cb4-26">fitted_model2 <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">workflow</span>(recipe2, lgbm_model) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">fit</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">data =</span> train_data)</span></code></pre></div></div>
</div>
</section>
<section id="results" class="level2">
<h2 class="anchored" data-anchor-id="results">Results</h2>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb5" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb5-1">results1 <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">bind_cols</span>(test_data, <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">predict</span>(fitted_model1, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">new_data =</span> test_data, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">type =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"prob"</span>))</span>
<span id="cb5-2">results2 <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">bind_cols</span>(test_data, <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">predict</span>(fitted_model2, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">new_data =</span> test_data, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">type =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"prob"</span>))</span>
<span id="cb5-3"></span>
<span id="cb5-4">roc_auc1 <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">roc_auc</span>(results1, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">truth =</span> target, .pred_a_1)</span>
<span id="cb5-5">roc_auc2 <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">roc_auc</span>(results2, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">truth =</span> target, .pred_a_1)</span>
<span id="cb5-6"></span>
<span id="cb5-7"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">bind_rows</span>(</span>
<span id="cb5-8">  <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"both features"</span> <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">=</span> roc_auc1,</span>
<span id="cb5-9">  <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"X_12m dropped"</span> <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">=</span> roc_auc2,</span>
<span id="cb5-10">  <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">.id =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"model"</span></span>
<span id="cb5-11">)</span>
<span id="cb5-12"><span class="do" style="color: #5E5E5E;
background-color: null;
font-style: italic;">## # A tibble: 2 × 4</span></span>
<span id="cb5-13"><span class="do" style="color: #5E5E5E;
background-color: null;
font-style: italic;">##   model         .metric .estimator .estimate</span></span>
<span id="cb5-14"><span class="do" style="color: #5E5E5E;
background-color: null;
font-style: italic;">##   &lt;chr&gt;         &lt;chr&gt;   &lt;chr&gt;          &lt;dbl&gt;</span></span>
<span id="cb5-15"><span class="do" style="color: #5E5E5E;
background-color: null;
font-style: italic;">## 1 both features roc_auc binary         0.650</span></span>
<span id="cb5-16"><span class="do" style="color: #5E5E5E;
background-color: null;
font-style: italic;">## 2 X_12m dropped roc_auc binary         0.543</span></span></code></pre></div></div>
</div>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb6" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb6-1">roc_plot <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">function</span>(results, auc, title) {</span>
<span id="cb6-2">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">roc_curve</span>(results, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">truth =</span> target, .pred_a_1) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb6-3">    <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">ggplot</span>(<span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">aes</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">x =</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span> <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">-</span> specificity, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">y =</span> sensitivity)) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span></span>
<span id="cb6-4">    <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">geom_path</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">linewidth =</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span>) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span></span>
<span id="cb6-5">    <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">geom_abline</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">lty =</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">3</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">linewidth =</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span>) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span></span>
<span id="cb6-6">    <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">labs</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">title =</span> title, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">x =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"1 - Specificity"</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">y =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Sensitivity"</span>) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span></span>
<span id="cb6-7">    <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">annotate</span>(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"text"</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">x =</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0.6</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">y =</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0.2</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">label =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">paste</span>(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"AUC ="</span>, <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">round</span>(auc<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span>.estimate, <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">2</span>)),</span>
<span id="cb6-8">             <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">size =</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">8</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">color =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"royalblue"</span>)</span>
<span id="cb6-9">}</span>
<span id="cb6-10"></span>
<span id="cb6-11">feat_imp_plot <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">function</span>(fitted_model) {</span>
<span id="cb6-12">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">lgb.importance</span>(<span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">extract_fit_engine</span>(fitted_model)) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb6-13">    <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">as_tibble</span>() <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb6-14">    <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">ggplot</span>(<span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">aes</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">x =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">reorder</span>(Feature, Gain), <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">y =</span> Gain)) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span></span>
<span id="cb6-15">    <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">geom_col</span>() <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span></span>
<span id="cb6-16">    <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">coord_flip</span>() <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span></span>
<span id="cb6-17">    <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">labs</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">title =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Feature importance"</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">x =</span> <span class="cn" style="color: #8f5902;
background-color: null;
font-style: inherit;">NULL</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">y =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Gain"</span>)</span>
<span id="cb6-18">}</span>
<span id="cb6-19"></span>
<span id="cb6-20">final_plot <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">roc_plot</span>(results1, roc_auc1, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"ROC curve: both features"</span>) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">feat_imp_plot</span>(fitted_model1) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span></span>
<span id="cb6-21">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">roc_plot</span>(results2, roc_auc2, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"ROC curve: X_12m dropped"</span>) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">feat_imp_plot</span>(fitted_model2) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span></span>
<span id="cb6-22">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">plot_layout</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">ncol =</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">2</span>)</span>
<span id="cb6-23">final_plot</span></code></pre></div></div>
<div class="cell-output-display">
<div>
<figure class="figure">
<p><img src="https://athospd.github.io/athosdamiani/posts/2026-11-20-feature-selection-by-correlation/index_files/figure-html/unnamed-chunk-6-1.png" class="img-fluid figure-img" width="960"></p>
</figure>
</div>
</div>
</div>
<p>With both features, the trees can split on <code>X_12m</code> and <code>X_06m</code> together and approximate their ratio, and the two features rank at the top of the importance. Without <code>X_12m</code>, the model sees <code>X_06m</code> alone, and the information in the ratio is gone. The AUROC drops from 0.65 to 0.54.</p>
</section>
<section id="what-to-check-instead" class="level2">
<h2 class="anchored" data-anchor-id="what-to-check-instead">What to check instead</h2>
<p>The correlation between two features tells you whether they carry the same values, not whether they carry the same information about the target. The rule works when the second feature is a noisy copy of the first. It fails when the signal is in how they differ.</p>
<p>Checking <img src="https://latex.codecogs.com/png.latex?%5Ctext%7Bcor%7D(X_1,%20f(X_1,%20X_2))"> for ratios, differences and the like helps, but only for the combinations you think of. A better question to ask is whether the model uses the feature at all, and how that compares to a feature that is pure noise.</p>
<p>That is the random probe idea. Add one or more random features, the probes, to the data, fit a model, and rank every feature by its importance. A probe can’t carry any information about the target, so its importance is the level a useless feature reaches by chance. Features that rank below the probes are candidates to drop. The method goes back to Stoppiglia, Dreyfus, Dubois and Oussar, <a href="https://www.jmlr.org/papers/v3/stoppiglia03a.html">Ranking a Random Feature for Variable and Feature Selection</a> (JMLR, 2003), which also gives a statistical stopping rule for where to cut the ranking.</p>
<div class="quarto-figure quarto-figure-center">
<figure class="figure">
<p><img src="https://athospd.github.io/athosdamiani/posts/2026-11-20-feature-selection-by-correlation/random-probes.png" class="img-fluid figure-img"></p>
<figcaption>The random probe strategy: add noise features, fit a tree model, rank features by importance, and drop those below the best probe.</figcaption>
</figure>
</div>
<p>A random forest is a good model for the ranking: it captures interactions such as the ratio here without you having to build them, and its feature importance comes for free. Ready-made tools:</p>
<ul>
<li>R: <a href="https://www.jstatsoft.org/v36/i11">{Boruta}</a> (Kursa and Rudnicki, <em>Journal of Statistical Software</em>, 2010) wraps a random forest. Its probes, the “shadow” features, are shuffled copies of each real feature. A feature is kept when its importance beats the best shadow more often than chance would allow, over many runs. Boruta looks for <em>all</em> relevant features, so it keeps redundant ones too: the opposite of the correlation filter.</li>
<li>Python: <a href="https://feature-engine.trainindata.com/en/latest/api_doc/selection/ProbeFeatureSelection.html"><code>ProbeFeatureSelection</code></a> from feature-engine adds random probe features, fits a scikit-learn model with cross-validation, and drops the features whose importance is below the probes’ (their mean, their maximum, or the mean plus 3 standard deviations).</li>
</ul>
<p>The simulation already has probes: <code>random1</code>, <code>random2</code> and <code>random3</code>. In the feature importance plot of the model with both features, <code>X_12m</code> and <code>X_06m</code> are far above all three, so the probe rule keeps both, which is what this data needs. The correlation filter would have dropped one of them.</p>
<p>The gist also has commented-out versions with the product, the difference and the sum in place of the ratio, if you want to try them.</p>
<p>Was this obvious to you? New? Or misleading in practice? I’d like to know.</p>


</section>

 ]]></description>
  <category>machine learning</category>
  <category>feature selection</category>
  <category>lightgbm</category>
  <category>tidymodels</category>
  <category>r</category>
  <guid>https://athospd.github.io/athosdamiani/posts/2026-11-20-feature-selection-by-correlation/</guid>
  <pubDate>Fri, 20 Nov 2026 00:00:00 GMT</pubDate>
  <media:content url="https://athospd.github.io/athosdamiani/posts/2026-11-20-feature-selection-by-correlation/feature-selection.png" medium="image" type="image/png" height="101" width="144"/>
</item>
<item>
  <title>Untuned vs tuned xgboost, one tree at a time</title>
  <dc:creator>Athos Damiani</dc:creator>
  <link>https://athospd.github.io/athosdamiani/posts/2026-11-20-xgboost-untuned-vs-tuned/</link>
  <description><![CDATA[ 





<p>A while ago I <a href="https://www.linkedin.com/posts/athosdamiani_rstats-python-datascience-share-6963297319974785024-xrKM/">posted on LinkedIn</a> an “untuned vs tuned xgboost fit” GIF. This post rebuilds it from the <a href="https://gist.github.com/Athospd/1295babba90315b4e5002262c24b08a0">original code</a>, with both models side by side.</p>
<p><img src="https://athospd.github.io/athosdamiani/posts/2026-11-20-xgboost-untuned-vs-tuned/xgboost.gif" class="img-fluid"></p>
<ul>
<li><strong>points</strong>: 1,000 noisy observations of a sine curve.</li>
<li><strong>purple</strong>: the true curve, <img src="https://latex.codecogs.com/png.latex?%5Csin(2%5Cpi%20x)">.</li>
<li><strong>orange</strong>: the model’s prediction after each boosting round. Each round adds one tree, and each tree adds a few steps to the curve.</li>
</ul>
<p>Both panels use the same data and the same algorithm. Only the hyperparameters differ. With the defaults, the curve takes big steps early on and then starts chasing the noise. With tuned hyperparameters, it moves to the sine in smaller steps and stays smooth.</p>
<section id="the-data" class="level2">
<h2 class="anchored" data-anchor-id="the-data">The data</h2>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb1" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb1-1"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">library</span>(tidyverse)</span>
<span id="cb1-2"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">library</span>(tidymodels)</span>
<span id="cb1-3"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">library</span>(gganimate)</span>
<span id="cb1-4"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">theme_set</span>(<span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">theme_minimal</span>(<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">16</span>))</span>
<span id="cb1-5"></span>
<span id="cb1-6"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">set.seed</span>(<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span>)</span>
<span id="cb1-7">dados <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">tibble</span>(</span>
<span id="cb1-8">  <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">x =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">runif</span>(<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1000</span>) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">-</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0.5</span>,</span>
<span id="cb1-9">  <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">y =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">sin</span>(x <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">*</span> pi <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">*</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">2</span>) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">rnorm</span>(<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1000</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">sd =</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0.3</span>)</span>
<span id="cb1-10">)</span></code></pre></div></div>
</div>
<p>The noise has standard deviation 0.3, so no model can get a root mean squared error (RMSE) below about 0.3 on new data.</p>
</section>
<section id="two-models" class="level2">
<h2 class="anchored" data-anchor-id="two-models">Two models</h2>
<p>The tuned model uses the hyperparameters from the gist. The untuned one keeps <code>boost_tree()</code>’s defaults: 15 trees, <code>tree_depth = 6</code> and <code>learn_rate = 0.3</code>. Both start from the same <code>base_score</code>, the mean of <code>y</code>.</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb2" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb2-1">trees <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">30</span></span>
<span id="cb2-2"></span>
<span id="cb2-3">tuned <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">boost_tree</span>(</span>
<span id="cb2-4">  <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">mode =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"regression"</span>,</span>
<span id="cb2-5">  <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">mtry =</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span>,</span>
<span id="cb2-6">  <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">trees =</span> trees,</span>
<span id="cb2-7">  <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">min_n =</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span>,</span>
<span id="cb2-8">  <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">tree_depth =</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">4</span>,</span>
<span id="cb2-9">  <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">learn_rate =</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0.1</span>,</span>
<span id="cb2-10">  <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">sample_size =</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span>,</span>
<span id="cb2-11">  <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">loss_reduction =</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0.1</span></span>
<span id="cb2-12">) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb2-13">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">set_engine</span>(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"xgboost"</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">base_score =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">mean</span>(dados<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span>y))</span>
<span id="cb2-14"></span>
<span id="cb2-15">untuned <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">boost_tree</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">mode =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"regression"</span>) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb2-16">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">set_engine</span>(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"xgboost"</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">base_score =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">mean</span>(dados<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span>y))</span>
<span id="cb2-17"></span>
<span id="cb2-18">ajustes <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">list</span>(</span>
<span id="cb2-19">  <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">untuned =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">fit</span>(untuned, y <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">~</span> x, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">data =</span> dados),</span>
<span id="cb2-20">  <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">tuned   =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">fit</span>(tuned, y <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">~</span> x, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">data =</span> dados)</span>
<span id="cb2-21">)</span></code></pre></div></div>
</div>
</section>
<section id="one-prediction-per-boosting-round" class="level2">
<h2 class="anchored" data-anchor-id="one-prediction-per-boosting-round">One prediction per boosting round</h2>
<p><code>iterationrange = c(1, step + 1)</code> tells xgboost to predict with only the first <code>step</code> trees. The gist used the older <code>ntreelimit = step - 1</code>, but <code>ntreelimit = 0</code> means “all trees”, so its first frame showed the final fit. Here step 0 is the starting point, the <code>base_score</code>, and steps past the last tree (the untuned model has 15) keep the full model.</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb3" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb3-1">dados_xgb <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> dados <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">select</span>(x) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">as.matrix</span>()</span>
<span id="cb3-2"></span>
<span id="cb3-3">predict_xgb <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">function</span>(ajuste, step) {</span>
<span id="cb3-4">  pred <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">if</span> (step <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">==</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">0</span>) {</span>
<span id="cb3-5">    <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">rep</span>(<span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">mean</span>(dados<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span>y), <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">nrow</span>(dados))</span>
<span id="cb3-6">  } <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">else</span> {</span>
<span id="cb3-7">    xgboost<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">:::</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">predict.xgb.Booster</span>(ajuste<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span>fit, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">newdata =</span> dados_xgb, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">iterationrange =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">c</span>(<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span>, <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">min</span>(step, ajuste<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span>fit<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span>niter) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span>))</span>
<span id="cb3-8">  }</span>
<span id="cb3-9">  dados <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">mutate</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">pred =</span> pred)</span>
<span id="cb3-10">}</span>
<span id="cb3-11"></span>
<span id="cb3-12">df <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">expand_grid</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">modelo =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">names</span>(ajustes), <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">step =</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">0</span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">:</span>trees) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb3-13">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">mutate</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">preds =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">map2</span>(modelo, step, <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">~</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">predict_xgb</span>(ajustes[[.x]], .y))) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb3-14">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">unnest</span>(preds) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb3-15">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">mutate</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">modelo =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">factor</span>(modelo, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">levels =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">names</span>(ajustes)))</span></code></pre></div></div>
</div>
</section>
<section id="the-gif" class="level2">
<h2 class="anchored" data-anchor-id="the-gif">The GIF</h2>
<p>Same plot as the gist, with one panel per model.</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb4" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb4-1">gif <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> df <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb4-2">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">ggplot</span>(<span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">aes</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">x =</span> x)) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span></span>
<span id="cb4-3">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">geom_point</span>(<span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">aes</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">y =</span> y), <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">size =</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">1.5</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">alpha =</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0.3</span>) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span></span>
<span id="cb4-4">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">stat_function</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">fun =</span> <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">~</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">sin</span>(. <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">*</span> pi <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">*</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">2</span>), <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">colour =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"purple"</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">linewidth =</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">1.5</span>) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span></span>
<span id="cb4-5">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">geom_step</span>(<span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">aes</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">y =</span> pred), <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">colour =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"orange"</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">linewidth =</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">1.5</span>) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span></span>
<span id="cb4-6">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">facet_wrap</span>(<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">~</span> modelo) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span></span>
<span id="cb4-7">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">transition_time</span>(step) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span></span>
<span id="cb4-8">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">labs</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">title =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">'Step: {frame_time}'</span>) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span></span>
<span id="cb4-9">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">ease_aes</span>(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"linear"</span>)</span>
<span id="cb4-10"></span>
<span id="cb4-11">anim <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">animate</span>(gif, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">nframes =</span> trees <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span> <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">10</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">fps =</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">5</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">end_pause =</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">10</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">width =</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1000</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">height =</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">450</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">units =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"px"</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">res =</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">100</span>)</span>
<span id="cb4-12"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">anim_save</span>(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"xgboost.gif"</span>, anim)</span></code></pre></div></div>
</div>
</section>
<section id="how-much-better" class="level2">
<h2 class="anchored" data-anchor-id="how-much-better">How much better?</h2>
<p>The GIF shows the difference. RMSE puts a number on it: on the training data, and on 5,000 new points from the same curve.</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb5" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb5-1"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">set.seed</span>(<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">2</span>)</span>
<span id="cb5-2">novos <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">tibble</span>(</span>
<span id="cb5-3">  <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">x =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">runif</span>(<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">5000</span>) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">-</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0.5</span>,</span>
<span id="cb5-4">  <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">y =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">sin</span>(x <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">*</span> pi <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">*</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">2</span>) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">rnorm</span>(<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">5000</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">sd =</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0.3</span>)</span>
<span id="cb5-5">)</span>
<span id="cb5-6"></span>
<span id="cb5-7">rmse_de <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">function</span>(ajuste, d) <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">rmse_vec</span>(d<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span>y, <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">predict</span>(ajuste, d)<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span>.pred)</span>
<span id="cb5-8"></span>
<span id="cb5-9"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">tibble</span>(</span>
<span id="cb5-10">  <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">modelo =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">names</span>(ajustes),</span>
<span id="cb5-11">  <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">rmse_treino =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">map_dbl</span>(ajustes, rmse_de, dados),</span>
<span id="cb5-12">  <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">rmse_novos  =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">map_dbl</span>(ajustes, rmse_de, novos)</span>
<span id="cb5-13">) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb5-14">  knitr<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">::</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">kable</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">digits =</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">3</span>)</span></code></pre></div></div>
<div class="cell-output-display">
<table class="caption-top table table-sm table-striped small">
<thead>
<tr class="header">
<th style="text-align: left;">modelo</th>
<th style="text-align: right;">rmse_treino</th>
<th style="text-align: right;">rmse_novos</th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td style="text-align: left;">untuned</td>
<td style="text-align: right;">0.241</td>
<td style="text-align: right;">0.322</td>
</tr>
<tr class="even">
<td style="text-align: left;">tuned</td>
<td style="text-align: right;">0.287</td>
<td style="text-align: right;">0.310</td>
</tr>
</tbody>
</table>
</div>
</div>
<p>The untuned model has the lower error on the training data, below the 0.3 noise level, which means it fitted part of the noise. On new data the order flips and the tuned model wins.</p>
</section>
<section id="tuning-for-real" class="level2">
<h2 class="anchored" data-anchor-id="tuning-for-real">Tuning for real</h2>
<p>The hyperparameters above came with the gist. In practice you find them with cross-validation, for example with {tune}:</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb6" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb6-1">spec <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">boost_tree</span>(</span>
<span id="cb6-2">  <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">mode =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"regression"</span>,</span>
<span id="cb6-3">  <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">trees =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">tune</span>(), <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">tree_depth =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">tune</span>(), <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">learn_rate =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">tune</span>(), <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">loss_reduction =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">tune</span>()</span>
<span id="cb6-4">) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb6-5">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">set_engine</span>(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"xgboost"</span>)</span>
<span id="cb6-6"></span>
<span id="cb6-7">resultados <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">tune_grid</span>(spec, y <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">~</span> x, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">resamples =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">vfold_cv</span>(dados, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">v =</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">5</span>), <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">grid =</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">20</span>)</span>
<span id="cb6-8"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">show_best</span>(resultados, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">metric =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"rmse"</span>)</span></code></pre></div></div>
</div>
<p>Defaults are a starting point, not a model. Tune your models, folks!</p>


</section>

 ]]></description>
  <category>machine learning</category>
  <category>xgboost</category>
  <category>tidymodels</category>
  <category>r</category>
  <category>animation</category>
  <guid>https://athospd.github.io/athosdamiani/posts/2026-11-20-xgboost-untuned-vs-tuned/</guid>
  <pubDate>Fri, 20 Nov 2026 00:00:00 GMT</pubDate>
  <media:content url="https://athospd.github.io/athosdamiani/posts/2026-11-20-xgboost-untuned-vs-tuned/xgboost.png" medium="image" type="image/png" height="65" width="144"/>
</item>
<item>
  <title>Top 10 packages for data science, 2026 edition</title>
  <dc:creator>Athos Damiani</dc:creator>
  <link>https://athospd.github.io/athosdamiani/posts/2026-11-20-top10-packages-2026/</link>
  <description><![CDATA[ 





<p>In 2017 I wrote my <a href="../2017-02-21-top10-pacotes-para-data-science/">top 10 packages for data science</a>. Nine years later, some of them are still on my list, some were replaced by their successors, and some new ones changed the way I work. Here is the 2026 edition, plus a special mention for the packages of the GenAI era.</p>
<p><img src="https://athospd.github.io/athosdamiani/posts/2026-11-20-top10-packages-2026/podium-2026.png" class="img-fluid"></p>
<section id="purrr-vector-and-list-manipulation" class="level3">
<h3 class="anchored" data-anchor-id="purrr-vector-and-list-manipulation"><a href="https://purrr.tidyverse.org/">10. purrr</a> (vector and list manipulation)</h3>
<p>Still here, and still putting the <code>for</code> loop at risk of extinction. <code>map()</code> and friends apply functions over vectors and lists without i, j, k indices, and they are the natural way to work with <em>list-columns</em>. Since version 1.1, <code>in_parallel()</code> runs the same <code>map()</code> calls in parallel, too.</p>
</section>
<section id="tidymodels-statistical-modeling" class="level3">
<h3 class="anchored" data-anchor-id="tidymodels-statistical-modeling"><a href="https://www.tidymodels.org/">9. tidymodels</a> (statistical modeling)</h3>
<p><em>Replaces caret.</em> It was made by caret’s own author, Max Kuhn, as its successor. The pieces of the modeling process are separate packages that fit together: {rsample} for train/test splits and cross-validation, {recipes} for preprocessing, {parsnip} for one interface to many engines (glmnet, ranger, xgboost, lightgbm…), {tune} for hyperparameter tuning, and {yardstick} for metrics. Most posts on this blog with a model in them use it, like the <a href="../2026-11-20-xgboost-untuned-vs-tuned/">untuned vs tuned xgboost</a> one.</p>
</section>
<section id="quarto-reports" class="level3">
<h3 class="anchored" data-anchor-id="quarto-reports"><a href="https://quarto.org/">8. Quarto</a> (reports)</h3>
<p><em>Replaces knitr/rmarkdown.</em> Quarto is RMarkdown’s successor, from the same team. Same idea, write the analysis and let the tool make the pdf, word or html, but it works with R, Python and Julia alike, and it does websites, books, slides and dashboards out of the box. knitr is still under the hood when the code is R.</p>
<p>PS: this blog is made with Quarto!</p>
</section>
<section id="mirai-async-and-parallel-computing" class="level3">
<h3 class="anchored" data-anchor-id="mirai-async-and-parallel-computing"><a href="https://mirai.r-lib.org/">7. mirai</a> (async and parallel computing)</h3>
<p><em>Replaces stringr.</em> {mirai} evaluates R code asynchronously and in parallel, on your laptop or spread across the machines of a cluster. It’s now an official cluster type in R’s own {parallel} package. Its best trick is that you rarely call it directly: it is the engine behind a lot of the R you already use.</p>
<ul>
<li>{purrr}: <code>map(x, in_parallel(f))</code> runs a <code>map()</code> in parallel.</li>
<li>{shiny}: long computations run in the background (<code>ExtendedTask</code>) while the app stays responsive.</li>
<li>{targets}: through {crew}, the steps of a pipeline run on parallel workers, locally or in the cloud.</li>
<li>{promises}: any async R code can wait on a mirai.</li>
</ul>
<p>Parallel R used to mean choosing between a handful of frameworks with different rules. Now it’s mostly one engine, and it’s fast.</p>
</section>
<section id="lubridate-date-manipulation" class="level3">
<h3 class="anchored" data-anchor-id="lubridate-date-manipulation"><a href="https://lubridate.tidyverse.org/">6. lubridate</a> (date manipulation)</h3>
<p>Dates are still an endless source of headaches, and lubridate is still the cure: parsing, arithmetic, comparisons, rounding, time zones.</p>
</section>
<section id="shiny-web-apps" class="level3">
<h3 class="anchored" data-anchor-id="shiny-web-apps"><a href="https://shiny.posit.co/">5. shiny</a> (web apps)</h3>
<p>Web applications without html, css or javascript. Shiny is more mature than ever, with modern themes from {bslib}, and it now exists for Python too.</p>
</section>
<section id="targets-pipelines" class="level3">
<h3 class="anchored" data-anchor-id="targets-pipelines"><a href="https://docs.ropensci.org/targets/">4. targets</a> (pipelines)</h3>
<p><em>New on the list.</em> {targets} turns an analysis into a pipeline of steps that know their dependencies. Change one step and only what depends on it runs again; everything else is skipped. For projects with a long data preparation or slow models, it’s the difference between waiting hours and waiting seconds, and the results are reproducible by design.</p>
</section>
<section id="tidyverse-data-manipulation" class="level3">
<h3 class="anchored" data-anchor-id="tidyverse-data-manipulation"><a href="https://www.tidyverse.org/">3. tidyverse</a> (data manipulation)</h3>
<p><em>Replaces dplyr (3rd) and tidyr (4th).</em> In 2017 they were two separate medals. Today <code>library(tidyverse)</code> loads them together with ggplot2, readr, tibble, purrr, stringr, forcats and lubridate, and that’s how most people use them. dplyr’s verbs (<code>filter()</code>, <code>arrange()</code>, <code>select()</code>, <code>distinct()</code>, <code>mutate()</code>, <code>summarise()</code>) are the same, now with <code>.by</code> for grouping in one line, and tidyr’s <code>gather()</code>/<code>spread()</code> became <code>pivot_longer()</code>/<code>pivot_wider()</code>.</p>
</section>
<section id="ggplot2-visualization" class="level3">
<h3 class="anchored" data-anchor-id="ggplot2-visualization"><a href="https://ggplot2.tidyverse.org/">2. ggplot2</a> (visualization)</h3>
<p>Still the silver medal and still the most famous R package. The <a href="https://vita.had.co.nz/papers/layered-grammar.pdf">grammar of graphics</a> is as useful as ever, and the ecosystem around it ({patchwork}, {gganimate}, {ggiraph} and many more) keeps growing. A must in any data scientist’s toolbox.</p>
</section>
<section id="torch-deep-learning" class="level3">
<h3 class="anchored" data-anchor-id="torch-deep-learning"><a href="https://torch.mlverse.org/">1. torch</a> (deep learning)</h3>
<p><em>Replaces magrittr.</em> The pipe was my gold medal in 2017. It won so completely that base R has had its own pipe, <code>|&gt;</code>, since R 4.1, so magrittr retires from the list with honors.</p>
<p>The gold medal now goes to {torch}: deep learning in R, built directly on the C++ library behind PyTorch (LibTorch), without Python. Tensors on the GPU, automatic differentiation and neural network modules, with an R interface. Automatic differentiation alone makes it worth learning: any model you can write as a function, you can fit with gradient descent. A few posts on this blog use it, like <a href="../2021-02-04-simple-audio-classification-with-torch/">simple audio classification</a> and the <a href="../2026-11-20-learning-rate/">learning rate</a> one.</p>
</section>
<section id="honorable-mentions" class="level3">
<h3 class="anchored" data-anchor-id="honorable-mentions">Honorable mentions</h3>
<ul>
<li><strong>keras3</strong> (deep learning with a high-level API)</li>
<li><strong>torchvision</strong> (images, datasets and pretrained models for torch)</li>
<li><strong>forcats</strong> (factor utilities)</li>
<li><strong>Rcpp</strong> (R to C++)</li>
<li><strong>FactoMineR</strong> (multivariate analysis)</li>
<li><strong>DBI</strong> (database connection)</li>
<li><strong>httr2/xml2/rvest</strong> (web tools)</li>
<li><strong>janitor</strong> (data cleaning for modeling)</li>
<li><strong>roxygen2</strong> (R package creation)</li>
<li><strong>devtools/usethis</strong> (package development tools)</li>
<li><strong>htmlwidgets</strong> (integration between R and JavaScript libraries)</li>
<li><strong>leaflet</strong> (interactive maps)</li>
</ul>
</section>
<section id="special-mention-the-genai-era" class="level3">
<h3 class="anchored" data-anchor-id="special-mention-the-genai-era">Special mention: the GenAI era</h3>
<p>LLMs, RAG and agents arrived in R too, and a new family of packages came with them:</p>
<ul>
<li><a href="https://ellmer.tidyverse.org/"><strong>ellmer</strong></a>: chat with LLMs from many providers (Claude, OpenAI, Gemini, local models via Ollama and more), with streaming, tool calling and structured data extraction.</li>
<li><a href="https://ragnar.tidyverse.org/"><strong>ragnar</strong></a>: Retrieval-Augmented Generation (RAG) workflows. Reads documents, splits them into chunks, generates embeddings, stores them and retrieves the relevant ones for the LLM.</li>
<li><a href="https://posit-dev.github.io/mcptools/"><strong>mcptools</strong></a>: the Model Context Protocol (MCP) for R. Serve R functions as tools for apps like Claude Desktop and Claude Code, or use other MCP servers’ tools from R.</li>
<li><a href="https://posit-dev.github.io/btw/"><strong>btw</strong></a>: describes your R objects, workspace and package documentation in a format LLMs understand.</li>
<li><a href="https://vitals.tidyverse.org/"><strong>vitals</strong></a>: evaluates LLM-based products, an R port of Python’s Inspect framework.</li>
<li><a href="https://posit-dev.github.io/shinychat/"><strong>shinychat</strong></a> and <a href="https://posit-dev.github.io/querychat/"><strong>querychat</strong></a>: a chat interface for Shiny apps, and a chat that turns users’ questions into SQL queries over your data.</li>
</ul>
<p>I’m still discovering and learning this whole new world. But it’s as sure as the sunrise that it’s here to stay.</p>
</section>
<section id="what-the-downloads-say" class="level2">
<h2 class="anchored" data-anchor-id="what-the-downloads-say">What the downloads say</h2>
<p>A top 10 is a matter of taste, but the download counts are data. The CRAN logs of Posit’s mirror are public, so here are the 2017 list and the 2026 list, with downloads in the month before each post.</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb1" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb1-1"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">library</span>(tidyverse)</span>
<span id="cb1-2"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">options</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">knitr.kable.NA =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">""</span>)</span>
<span id="cb1-3"></span>
<span id="cb1-4">downloads <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">function</span>(pkgs, from, to) {</span>
<span id="cb1-5">  url <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">str_glue</span>(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"https://cranlogs.r-pkg.org/downloads/total/{from}:{to}/{str_flatten(pkgs, ',')}"</span>)</span>
<span id="cb1-6">  jsonlite<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">::</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">fromJSON</span>(url) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">|&gt;</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">select</span>(package, downloads)</span>
<span id="cb1-7">}</span>
<span id="cb1-8"></span>
<span id="cb1-9">top_2017 <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">c</span>(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"magrittr"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"ggplot2"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"dplyr"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"tidyr"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"shiny"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"lubridate"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"stringr"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"rmarkdown"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"caret"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"purrr"</span>)</span>
<span id="cb1-10">top_2026 <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">c</span>(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"torch"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"ggplot2"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"tidyverse"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"targets"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"shiny"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"lubridate"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"mirai"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"quarto"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"tidymodels"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"purrr"</span>)</span>
<span id="cb1-11"></span>
<span id="cb1-12">pkgs <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">union</span>(top_2017, top_2026)</span>
<span id="cb1-13"></span>
<span id="cb1-14"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">bind_rows</span>(</span>
<span id="cb1-15">  <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">feb_2017 =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">downloads</span>(pkgs, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"2017-01-21"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"2017-02-20"</span>),</span>
<span id="cb1-16">  <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">nov_2026 =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">downloads</span>(pkgs, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"2026-10-20"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"2026-11-18"</span>),</span>
<span id="cb1-17">  <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">.id =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"month"</span></span>
<span id="cb1-18">) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">|&gt;</span></span>
<span id="cb1-19">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">pivot_wider</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">names_from =</span> month, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">values_from =</span> downloads) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">|&gt;</span></span>
<span id="cb1-20">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">mutate</span>(</span>
<span id="cb1-21">    <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">feb_2017 =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">na_if</span>(feb_2017, <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">0</span>), <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># torch, targets, tidymodels, quarto and mirai didn't exist yet</span></span>
<span id="cb1-22">    <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">rank_2017 =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">match</span>(package, top_2017),</span>
<span id="cb1-23">    <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">rank_2026 =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">match</span>(package, top_2026)</span>
<span id="cb1-24">  ) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">|&gt;</span></span>
<span id="cb1-25">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">arrange</span>(rank_2026, rank_2017) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">|&gt;</span></span>
<span id="cb1-26">  knitr<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">::</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">kable</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">format.args =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">list</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">big.mark =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">","</span>))</span></code></pre></div></div>
<div class="cell-output-display">
<table class="caption-top table table-sm table-striped small">
<thead>
<tr class="header">
<th style="text-align: left;">package</th>
<th style="text-align: right;">feb_2017</th>
<th style="text-align: right;">nov_2026</th>
<th style="text-align: right;">rank_2017</th>
<th style="text-align: right;">rank_2026</th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td style="text-align: left;">torch</td>
<td style="text-align: right;"></td>
<td style="text-align: right;">43,751</td>
<td style="text-align: right;"></td>
<td style="text-align: right;">1</td>
</tr>
<tr class="even">
<td style="text-align: left;">ggplot2</td>
<td style="text-align: right;">437,680</td>
<td style="text-align: right;">2,115,852</td>
<td style="text-align: right;">2</td>
<td style="text-align: right;">2</td>
</tr>
<tr class="odd">
<td style="text-align: left;">tidyverse</td>
<td style="text-align: right;">49,273</td>
<td style="text-align: right;">864,499</td>
<td style="text-align: right;"></td>
<td style="text-align: right;">3</td>
</tr>
<tr class="even">
<td style="text-align: left;">targets</td>
<td style="text-align: right;"></td>
<td style="text-align: right;">12,599</td>
<td style="text-align: right;"></td>
<td style="text-align: right;">4</td>
</tr>
<tr class="odd">
<td style="text-align: left;">shiny</td>
<td style="text-align: right;">148,314</td>
<td style="text-align: right;">919,605</td>
<td style="text-align: right;">5</td>
<td style="text-align: right;">5</td>
</tr>
<tr class="even">
<td style="text-align: left;">lubridate</td>
<td style="text-align: right;">91,281</td>
<td style="text-align: right;">602,683</td>
<td style="text-align: right;">6</td>
<td style="text-align: right;">6</td>
</tr>
<tr class="odd">
<td style="text-align: left;">mirai</td>
<td style="text-align: right;"></td>
<td style="text-align: right;">41,213</td>
<td style="text-align: right;"></td>
<td style="text-align: right;">7</td>
</tr>
<tr class="even">
<td style="text-align: left;">quarto</td>
<td style="text-align: right;"></td>
<td style="text-align: right;">56,386</td>
<td style="text-align: right;"></td>
<td style="text-align: right;">8</td>
</tr>
<tr class="odd">
<td style="text-align: left;">tidymodels</td>
<td style="text-align: right;"></td>
<td style="text-align: right;">98,256</td>
<td style="text-align: right;"></td>
<td style="text-align: right;">9</td>
</tr>
<tr class="even">
<td style="text-align: left;">purrr</td>
<td style="text-align: right;">63,867</td>
<td style="text-align: right;">1,205,141</td>
<td style="text-align: right;">10</td>
<td style="text-align: right;">10</td>
</tr>
<tr class="odd">
<td style="text-align: left;">magrittr</td>
<td style="text-align: right;">355,815</td>
<td style="text-align: right;">1,688,325</td>
<td style="text-align: right;">1</td>
<td style="text-align: right;"></td>
</tr>
<tr class="even">
<td style="text-align: left;">dplyr</td>
<td style="text-align: right;">296,973</td>
<td style="text-align: right;">1,754,169</td>
<td style="text-align: right;">3</td>
<td style="text-align: right;"></td>
</tr>
<tr class="odd">
<td style="text-align: left;">tidyr</td>
<td style="text-align: right;">144,913</td>
<td style="text-align: right;">1,106,347</td>
<td style="text-align: right;">4</td>
<td style="text-align: right;"></td>
</tr>
<tr class="even">
<td style="text-align: left;">stringr</td>
<td style="text-align: right;">342,351</td>
<td style="text-align: right;">1,187,180</td>
<td style="text-align: right;">7</td>
<td style="text-align: right;"></td>
</tr>
<tr class="odd">
<td style="text-align: left;">rmarkdown</td>
<td style="text-align: right;">208,673</td>
<td style="text-align: right;">1,371,005</td>
<td style="text-align: right;">8</td>
<td style="text-align: right;"></td>
</tr>
<tr class="even">
<td style="text-align: left;">caret</td>
<td style="text-align: right;">87,426</td>
<td style="text-align: right;">132,869</td>
<td style="text-align: right;">9</td>
<td style="text-align: right;"></td>
</tr>
</tbody>
</table>
</div>
</div>
<p>A few things stand out:</p>
<ul>
<li>ggplot2 is the most downloaded package on CRAN that you’d install on purpose. Only rlang, a low-level dependency of almost everything, beats it.</li>
<li>magrittr still has more than 1.6 million downloads a month, mostly as a dependency of other packages, even with the native pipe in base R.</li>
<li>torch, mirai and targets are far from the top in downloads. They’re on my list because of what they do for my work, not because everyone uses them (yet).</li>
<li>Quarto is a command-line tool, not an R package. The <code>quarto</code> R package only calls it, so its downloads say little about how many people use Quarto.</li>
</ul>
<p>These numbers only count Posit’s CRAN mirror, so they are a sample, not the whole picture.</p>
<p>What about you, do you agree? Tell me which packages live in your heart! =)</p>


</section>

 ]]></description>
  <category>top 10</category>
  <category>packages</category>
  <category>r</category>
  <guid>https://athospd.github.io/athosdamiani/posts/2026-11-20-top10-packages-2026/</guid>
  <pubDate>Fri, 20 Nov 2026 00:00:00 GMT</pubDate>
  <media:content url="https://athospd.github.io/athosdamiani/posts/2026-11-20-top10-packages-2026/podium-2026.png" medium="image" type="image/png" height="81" width="144"/>
</item>
<item>
  <title>Querying Parquet with SQL from R, no warehouse needed</title>
  <dc:creator>Athos Damiani</dc:creator>
  <link>https://athospd.github.io/athosdamiani/posts/2026-11-17-parquet-sql-from-r/</link>
  <description><![CDATA[ 





<p>A lot of data engineering work is just <em>moving tables around and aggregating them</em>. You don’t always need a cluster for that. <a href="https://duckdb.org">DuckDB</a> is an in-process analytical database that reads Parquet directly, and with <code>dbplyr</code> you can write dplyr code that gets translated to SQL.</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb1" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb1-1"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">library</span>(DBI)</span>
<span id="cb1-2"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">library</span>(dplyr)</span>
<span id="cb1-3"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">library</span>(dbplyr)</span>
<span id="cb1-4"></span>
<span id="cb1-5">con <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">dbConnect</span>(duckdb<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">::</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">duckdb</span>())</span></code></pre></div></div>
</div>
<section id="create-some-raw-data-as-parquet" class="level2">
<h2 class="anchored" data-anchor-id="create-some-raw-data-as-parquet">Create some “raw” data as Parquet</h2>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb2" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb2-1"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">set.seed</span>(<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">42</span>)</span>
<span id="cb2-2">transactions <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">data.frame</span>(</span>
<span id="cb2-3">  <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">id       =</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">:</span><span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">10000</span>,</span>
<span id="cb2-4">  <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">customer =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">sample</span>(<span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">sprintf</span>(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"C%03d"</span>, <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">:</span><span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">200</span>), <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">10000</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">replace =</span> <span class="cn" style="color: #8f5902;
background-color: null;
font-style: inherit;">TRUE</span>),</span>
<span id="cb2-5">  <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">amount   =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">round</span>(<span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">rexp</span>(<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">10000</span>, <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span> <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">/</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">80</span>), <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">2</span>),</span>
<span id="cb2-6">  <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">day      =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">as.Date</span>(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"2026-01-01"</span>) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">sample</span>(<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">0</span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">:</span><span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">89</span>, <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">10000</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">replace =</span> <span class="cn" style="color: #8f5902;
background-color: null;
font-style: inherit;">TRUE</span>)</span>
<span id="cb2-7">)</span>
<span id="cb2-8"></span>
<span id="cb2-9">path <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">tempfile</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">fileext =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">".parquet"</span>)</span>
<span id="cb2-10">duckdb<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">::</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">duckdb_register</span>(con, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"tx_df"</span>, transactions)</span>
<span id="cb2-11"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">dbExecute</span>(con, <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">sprintf</span>(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"COPY tx_df TO '%s' (FORMAT PARQUET)"</span>, path))</span></code></pre></div></div>
<div class="cell-output cell-output-stdout">
<pre><code>[1] 10000</code></pre>
</div>
</div>
</section>
<section id="query-the-file-lazily-with-dplyr" class="level2">
<h2 class="anchored" data-anchor-id="query-the-file-lazily-with-dplyr">Query the file lazily with dplyr</h2>
<p><code>tbl()</code> points at the parquet file, nothing is loaded into R yet.</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb4" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb4-1">tx <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">tbl</span>(con, <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">sql</span>(<span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">sprintf</span>(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"SELECT * FROM read_parquet('%s')"</span>, path)))</span>
<span id="cb4-2"></span>
<span id="cb4-3">monthly <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> tx <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">|&gt;</span></span>
<span id="cb4-4">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">mutate</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">month =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">date_trunc</span>(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"month"</span>, day)) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">|&gt;</span></span>
<span id="cb4-5">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">group_by</span>(month) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">|&gt;</span></span>
<span id="cb4-6">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">summarise</span>(</span>
<span id="cb4-7">    <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">n_tx      =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">n</span>(),</span>
<span id="cb4-8">    <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">customers =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">n_distinct</span>(customer),</span>
<span id="cb4-9">    <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">revenue   =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">sum</span>(amount, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">na.rm =</span> <span class="cn" style="color: #8f5902;
background-color: null;
font-style: inherit;">TRUE</span>)</span>
<span id="cb4-10">  ) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">|&gt;</span></span>
<span id="cb4-11">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">arrange</span>(month)</span></code></pre></div></div>
</div>
<p>This is the SQL dbplyr generated:</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb5" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb5-1"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">show_query</span>(monthly)</span></code></pre></div></div>
<div class="cell-output cell-output-stdout">
<pre><code>&lt;SQL&gt;
SELECT
  "month",
  COUNT(*) AS n_tx,
  COUNT(DISTINCT row(customer)) AS customers,
  SUM(amount) AS revenue
FROM (
  SELECT q01.*, date_trunc('month', "day") AS "month"
  FROM (SELECT * FROM read_parquet('C:\Users\ap_da\AppData\Local\Temp\RtmpmYMvdD\file14b686d74f70.parquet')) q01
) q01
GROUP BY "month"
ORDER BY "month"</code></pre>
</div>
</div>
<p>And only now do we pull the (small) result into R:</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb7" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb7-1"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">collect</span>(monthly)</span></code></pre></div></div>
<div class="cell-output cell-output-stdout">
<pre><code># A tibble: 3 × 4
  month                n_tx customers revenue
  &lt;dttm&gt;              &lt;dbl&gt;     &lt;dbl&gt;   &lt;dbl&gt;
1 2026-01-01 00:00:00  3467       200 283743.
2 2026-02-01 00:00:00  3118       200 248429.
3 2026-03-01 00:00:00  3415       200 275635.</code></pre>
</div>
</div>
<p>The same pattern scales up: swap DuckDB for Spark (<code>sparklyr</code>) or Databricks SQL and the dplyr code stays the same.</p>


</section>

 ]]></description>
  <category>r</category>
  <category>duckdb</category>
  <category>sql</category>
  <category>parquet</category>
  <guid>https://athospd.github.io/athosdamiani/posts/2026-11-17-parquet-sql-from-r/</guid>
  <pubDate>Tue, 17 Nov 2026 00:00:00 GMT</pubDate>
  <media:content url="https://athospd.github.io/athosdamiani/posts/2026-11-17-parquet-sql-from-r/parquet-sql.png" medium="image" type="image/png" height="81" width="144"/>
</item>
<item>
  <title>Neural Style Transfer with Torch</title>
  <dc:creator>Athos Damiani</dc:creator>
  <link>https://athospd.github.io/athosdamiani/posts/2021-02-22-neural-style-transfer/</link>
  <description><![CDATA[ 





<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb1" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb1-1"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">library</span>(torch)</span>
<span id="cb1-2"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">library</span>(torchvision)</span>
<span id="cb1-3"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">library</span>(zeallot)</span>
<span id="cb1-4">device <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">torch_device</span>(<span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">if</span>(<span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">cuda_is_available</span>()) <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"cuda"</span> <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">else</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"cpu"</span>)</span>
<span id="cb1-5">cpu <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">torch_device</span>(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"cpu"</span>)</span></code></pre></div></div>
</div>
<p>Neural Style Transfer is one of the most fun and “artistic” techniques in deep learning. The image below sums up what we are going to do.</p>
<div class="quarto-figure quarto-figure-center">
<figure class="figure">
<p><img src="https://athospd.github.io/athosdamiani/posts/2021-02-22-neural-style-transfer/nst.webp" class="img-fluid figure-img"></p>
<figcaption>content</figcaption>
</figure>
</div>
<p>You give the neural network two images, <code>style</code> and <code>content</code>, and the result is the <code>content</code> image in the style of <code>style</code>. It’s like an Instagram filter, but with the style of your favorite artist =P.</p>
<p>This post adapts to R + torch an exercise from the <a href="https://www.coursera.org/learn/convolutional-neural-networks/">deeplearning.ai ‘Convolutional Neural Networks’ course</a> that I took, originally in Python + tensorflow. When I started writing this post, I meant to make it as didactic as possible, but I don’t think I could beat the Coursera course, so I’ll stick to the main points and comment the code for anyone who wants to create their own images! Also, <a href="https://www.youtube.com/watch?v=iAyL5iCWWAs">this video (in Portuguese) gives a summary of the NST strategy.</a></p>
<p>PS: while writing this post I found out that Daniel had already written <a href="https://torchvision.mlverse.org/articles/examples/style-transfer.html">a little NST example for torchvision.</a> I stole part of the code from there haha.</p>
<section id="key-concepts-and-ideas" class="level2">
<h2 class="anchored" data-anchor-id="key-concepts-and-ideas">Key concepts and ideas</h2>
<p>The idea is to generate a <code>generated</code> image G whose content is similar to <code>content</code> C and whose style is similar to <code>style</code> S. Content and style are usually extracted from a pre-trained convnet. The original paper <span class="citation" data-cites="gatys2015neural">(Gatys et al. 2015)</span> uses VGG19, which already comes with {torchvision}.</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb2" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb2-1"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># VGG19 model</span></span>
<span id="cb2-2">vgg <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">model_vgg19</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">pretrained =</span> <span class="cn" style="color: #8f5902;
background-color: null;
font-style: inherit;">TRUE</span>)<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span>features<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">to</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">device =</span> device)</span>
<span id="cb2-3"></span>
<span id="cb2-4"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># freezing the weights</span></span>
<span id="cb2-5">vgg<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span>parameters <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span> purrr<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">::</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">walk</span>(<span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">function</span>(param) param<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">requires_grad_</span>(<span class="cn" style="color: #8f5902;
background-color: null;
font-style: inherit;">FALSE</span>))</span></code></pre></div></div>
</div>
<p>So VGG19 will give us the features, and now we need to define cost functions to find the image with the best trade-off between the content of one image and the style of another. Building the algorithm can be split into three parts:</p>
<ul>
<li><code>content</code> cost function: <img src="https://latex.codecogs.com/png.latex?L_%7Bcontent%7D(C,%20G)"></li>
<li><code>style</code> cost function: <img src="https://latex.codecogs.com/png.latex?L_%7Bstyle%7D(S,%20G)"></li>
<li>Overall cost function: <img src="https://latex.codecogs.com/png.latex?L(G)%20=%20%5Calpha%20L_%7Bcontent%7D(C,%20G)%20+%20%5Cbeta%20L_%7Bstyle%7D(S,%20G)"></li>
</ul>
<section id="content-cost-function" class="level3">
<h3 class="anchored" data-anchor-id="content-cost-function"><code>content</code> cost function</h3>
<p>On choosing shallow versus deep layers:</p>
<ul>
<li>The first layers of a convolutional network tend to extract more basic patterns, such as edges and simple textures.</li>
<li>Deeper layers usually capture more sophisticated features, such as detailed textures and objects.</li>
</ul>
<p>As for picking a middle layer: we want the <code>generated</code> image to have content similar to <code>content</code>. The strategy is:</p>
<ol type="1">
<li>choose one layer of the network to represent this “content”.</li>
<li>take that layer for each of the <code>content</code> and <code>generated</code> images.</li>
<li>make the network force these two layers to be as similar as possible.</li>
</ol>
<p>So the cost function that reflects point (3) can simply be the mean squared error between the pixels of that layer:</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb3" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb3-1">content_loss <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">function</span>(content_layer, generated_layer) {</span>
<span id="cb3-2">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">nnf_mse_loss</span>(content_layer, generated_layer)</span>
<span id="cb3-3">}</span></code></pre></div></div>
</div>
<p>NOTE: In practice, you’ll get the “coolest” result if you <strong>choose layers from the middle</strong>: not too shallow, not too deep. Since VGG19 has 37 layers, the chosen one could be, for example, layer 14.</p>
</section>
<section id="style-cost-function" class="level3">
<h3 class="anchored" data-anchor-id="style-cost-function"><code>style</code> cost function</h3>
<p>The purpose of the <code>style</code> cost function is to minimize the distance between the so-called <em>Gram matrices</em> of the <code>style</code> and <code>generated</code> images.</p>
<section id="gram-matrix" class="level4">
<h4 class="anchored" data-anchor-id="gram-matrix">Gram matrix</h4>
<p>In the literature, the style matrix is called the <em>Gram matrix</em>. In mathematics, given a set of vectors, the <em>Gram matrix</em> is the matrix of the inner products of every pair of those vectors. It’s like a correlation matrix, but without centering by the mean. In practice, you take a layer of the network, reshape it into a 2D tensor (a matrix) and compute <img src="https://latex.codecogs.com/png.latex?A%20*%20A%5ET">.</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb4" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb4-1">gram_matrix <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">function</span>(tensor) {</span>
<span id="cb4-2">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">c</span>(b,c,h,w) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&lt;-%</span> tensor<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">size</span>()</span>
<span id="cb4-3">  tensor <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> tensor<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">view</span>(<span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">c</span>(c, h<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">*</span>w))</span>
<span id="cb4-4">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">torch_matmul</span>(tensor, tensor<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">t</span>())<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">/</span>(h<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">*</span>w)</span>
<span id="cb4-5">}</span></code></pre></div></div>
</div>
<p>The cost function <img src="https://latex.codecogs.com/png.latex?L_%7Bstyle%7D(S,%20G)"> is the good old mean squared error between the <em>Gram matrices</em>.</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb5" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb5-1">style_loss <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">function</span>(style_layer, generated_layer) {</span>
<span id="cb5-2">  style_gram <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">gram_matrix</span>(style_layer)</span>
<span id="cb5-3">  generated_gram <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">gram_matrix</span>(generated_layer)</span>
<span id="cb5-4">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">nnf_mse_loss</span>(style_gram, generated_gram)</span>
<span id="cb5-5">}</span></code></pre></div></div>
</div>
</section>
<section id="style-layers" class="level4">
<h4 class="anchored" data-anchor-id="style-layers">Style layers</h4>
<p>Instead of one, we take five intermediate layers of the network and compute the distances between their <em>Gram matrices</em>. The cost function then becomes a weighted sum of these five distances: <img src="https://latex.codecogs.com/png.latex?L_%7Bstyle%7D(S,%20G)%20=%20%5Csum_%7Bl=1%7D%5E%7B5%7D%5Clambda%5E%7B%5Bl%5D%7DL_%7Bstyle%7D%5E%7B%5Bl%5D%7D(S,G)"></p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb6" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb6-1">style_layers <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">c</span>(<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">2</span>, <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">7</span>, <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">12</span>, <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">21</span>, <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">29</span>) <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># choice of VGG layers</span></span>
<span id="cb6-2">lambdas <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">1e5</span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">/</span>(<span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">c</span>(<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span>,<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">10</span>,<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">10</span>,<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">10</span>,<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">10</span>)<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">^</span><span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">2</span>) <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># weights for each layer in the final style</span></span></code></pre></div></div>
</div>
<p>Now it’s worth creating an nn_module() that returns the intermediate layers of the network (VGG19, in this case). The <code>layers_out</code> argument lets us choose which layers we want back.</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb7" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb7-1">features <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">nn_module</span>(</span>
<span id="cb7-2">  <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">initialize =</span> <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">function</span>(convnet) {</span>
<span id="cb7-3">    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># could be any pre-trained convnet. We'll use VGG19</span></span>
<span id="cb7-4">    self<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span>convnet <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> convnet</span>
<span id="cb7-5">  },</span>
<span id="cb7-6">  <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">forward =</span> <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">function</span>(img, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">layers_out =</span> <span class="cn" style="color: #8f5902;
background-color: null;
font-style: inherit;">NULL</span>) {</span>
<span id="cb7-7">    layers <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">seq_along</span>(self<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span>convnet) <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 1 to 37</span></span>
<span id="cb7-8">    <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">if</span>(<span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">is.null</span>(layers_out)) layers_out <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> layers</span>
<span id="cb7-9">    conv_outs <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> purrr<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">::</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">accumulate</span>(layers, <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">~</span>self<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span>convnet[[.y]](.x), <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">.init =</span> img) <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># list of 37 tensors</span></span>
<span id="cb7-10">    conv_outs[layers_out] <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># list with only the selected layers</span></span>
<span id="cb7-11">  }</span>
<span id="cb7-12">)</span></code></pre></div></div>
</div>
</section>
</section>
</section>
<section id="optimization" class="level2">
<h2 class="anchored" data-anchor-id="optimization">Optimization</h2>
<p>Below is the commented code to run the optimization.</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb8" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb8-1"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># helper functions</span></span>
<span id="cb8-2">to_r <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">function</span>(x) <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">as.numeric</span>(x<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">to</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">device =</span> cpu))</span>
<span id="cb8-3"></span>
<span id="cb8-4">plot_image <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">function</span>(tensor) {</span>
<span id="cb8-5">  im <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> tensor<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">to</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">device =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"cpu"</span>)[<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span>,..]<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span></span>
<span id="cb8-6">    <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">permute</span>(<span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">c</span>(<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">2</span>, <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">3</span>, <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span>))<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span></span>
<span id="cb8-7">    <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">to</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">device =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"cpu"</span>)<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span></span>
<span id="cb8-8">    <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">clamp</span>(<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">0</span>,<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span>) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span> <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># make it [0,1]</span></span>
<span id="cb8-9">    <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">as.array</span>()</span>
<span id="cb8-10">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">par</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">mar =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">c</span>(<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">0</span>,<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">0</span>,<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">0</span>,<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">0</span>))</span>
<span id="cb8-11">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">plot</span>(<span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">as.raster</span>(im))</span>
<span id="cb8-12">}</span>
<span id="cb8-13"></span>
<span id="cb8-14">load_image <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">function</span>(path, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">geometry =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"250x200"</span>) {</span>
<span id="cb8-15">  img <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> path <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb8-16">    <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">magick_loader</span>() <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb8-17">    magick<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">::</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">image_resize</span>(geometry) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb8-18">    <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">transform_to_tensor</span>() <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb8-19">    <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">torch_unsqueeze</span>(<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span>)</span>
<span id="cb8-20"></span>
<span id="cb8-21">  img<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">to</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">device =</span> device)</span>
<span id="cb8-22">}</span></code></pre></div></div>
</div>
<p>The parameters and inputs worth experimenting with are:</p>
<ul>
<li><code>content</code> and <code>style</code>: the input images, one for content and one for style.</li>
<li><code>content_layer</code>: the VGG19 layer used to extract the content of the content image.</li>
<li><code>style_layers</code>: the VGG19 layers used to extract the style of the style image.</li>
<li><code>lambdas</code>: the weight of each style feature in the optimization.</li>
<li><code>content_weight</code>: the weight of the content features in the optimization.</li>
<li><code>style_weight</code>: the (overall) weight of the style features in the optimization.</li>
</ul>
<p>Two other parameters that drastically affect the result are the dimensions of the two input images. In the code below, <code>"400x400"</code> and <code>"350x500"</code> can be swapped for other dimensions to get different results.</p>
<p>While creating, you’ll be tweaking these parameters all the time!</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb9" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb9-1"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># INPUT: content and style images</span></span>
<span id="cb9-2">content <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">load_image</span>(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"content/posts/2021-02-22-neural-style-transfer/cristoredentor3.jpg"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"400x400"</span>)</span>
<span id="cb9-3">style <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">load_image</span>(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"content/posts/2021-02-22-neural-style-transfer/vangogh_starry_night.jpg"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"350x500"</span>)</span>
<span id="cb9-4"></span>
<span id="cb9-5"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># style and content feature setup</span></span>
<span id="cb9-6">content_layer <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">14</span></span>
<span id="cb9-7">style_layers <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">c</span>(<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">2</span>, <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">7</span>, <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">12</span>, <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">21</span>, <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">29</span>)</span>
<span id="cb9-8">lambdas <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">1e5</span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">/</span>(<span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">c</span>(<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span>,<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">10</span>,<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">10</span>,<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">10</span>,<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">10</span>)<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">^</span><span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">2</span>)</span>
<span id="cb9-9">content_weight <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">2</span></span>
<span id="cb9-10">style_weight <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">4e-1</span></span></code></pre></div></div>
</div>
<div class="quarto-figure quarto-figure-center">
<figure class="figure">
<p><img src="https://athospd.github.io/athosdamiani/posts/2021-02-22-neural-style-transfer/cristoredentor3.webp" class="img-fluid figure-img"></p>
<figcaption>content</figcaption>
</figure>
</div>
<div class="quarto-figure quarto-figure-center">
<figure class="figure">
<p><img src="https://athospd.github.io/athosdamiani/posts/2021-02-22-neural-style-transfer/vangogh_starry_night.webp" class="img-fluid figure-img"></p>
<figcaption>style</figcaption>
</figure>
</div>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb10" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb10-1"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># FEATURES: extracted from VGG19</span></span>
<span id="cb10-2">vgg_features <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">features</span>(vgg)</span>
<span id="cb10-3">content_features <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">vgg_features</span>(content, content_layer)</span>
<span id="cb10-4">style_features <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">vgg_features</span>(style, style_layers)</span>
<span id="cb10-5"></span>
<span id="cb10-6"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># OUTPUT: generated image</span></span>
<span id="cb10-7">generated <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">torch_clone</span>(content)<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">requires_grad_</span>(<span class="cn" style="color: #8f5902;
background-color: null;
font-style: inherit;">TRUE</span>)</span>
<span id="cb10-8">optim <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">optim_adam</span>(generated, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">lr =</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0.02</span>)</span>
<span id="cb10-9">lr_scheduler <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">lr_step</span>(optim, <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">100</span>, <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0.9</span>)</span>
<span id="cb10-10"></span>
<span id="cb10-11"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># optimization loop</span></span>
<span id="cb10-12"><span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">for</span>(step <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">in</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">seq_len</span>(<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">8000</span>)) {</span>
<span id="cb10-13">  optim<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">zero_grad</span>()</span>
<span id="cb10-14">  <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># update the features of the image being generated</span></span>
<span id="cb10-15">  generated_features <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">vgg_features</span>(generated, <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">c</span>(content_layer, style_layers))</span>
<span id="cb10-16"></span>
<span id="cb10-17">  <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># losses</span></span>
<span id="cb10-18">  LC <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">content_loss</span>(content_features[[<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span>]], generated_features[[<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span>]])</span>
<span id="cb10-19">  LS <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">0</span></span>
<span id="cb10-20">  <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">for</span>(i <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">in</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">seq_along</span>(lambdas))</span>
<span id="cb10-21">    LS <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> LS <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span> lambdas[i]<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">*</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">style_loss</span>(style_features[[i]], generated_features[<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">-</span><span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span>][[i]])</span>
<span id="cb10-22"></span>
<span id="cb10-23">  loss <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> content_weight <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">*</span> LC <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span> style_weight <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">*</span> LS</span>
<span id="cb10-24"></span>
<span id="cb10-25">  loss<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">backward</span>()</span>
<span id="cb10-26">  optim<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">step</span>()</span>
<span id="cb10-27">  lr_scheduler<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">step</span>()</span>
<span id="cb10-28"></span>
<span id="cb10-29">  <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># feedback</span></span>
<span id="cb10-30">  <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">if</span>(step <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%%</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">100</span> <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">==</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">0</span>) {</span>
<span id="cb10-31">    <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">cat</span>(glue<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">::</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">glue</span>(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"LC = {to_r(LC)} - LS = {to_r(LS)} - Loss = {to_r(loss)}</span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">\n\n</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"</span>))</span>
<span id="cb10-32">    <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">plot_image</span>(generated)</span>
<span id="cb10-33">  }</span>
<span id="cb10-34">}</span>
<span id="cb10-35"></span>
<span id="cb10-36"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># final image</span></span>
<span id="cb10-37"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">plot_image</span>(generated)</span></code></pre></div></div>
</div>
<pre><code>LC = 2.70741701126099 - LS = 36.7998428344727 - Loss = 20.1347713470459
LC = 2.61282753944397 - LS = 24.3190288543701 - Loss = 14.9532661437988
LC = 2.54735898971558 - LS = 19.7909469604492 - Loss = 13.0110969543457
LC = 2.49572896957397 - LS = 16.9353790283203 - Loss = 11.7656097412109
LC = 2.44669985771179 - LS = 16.6177845001221 - Loss = 11.5405139923096</code></pre>
<p><img src="https://athospd.github.io/athosdamiani/posts/2021-02-22-neural-style-transfer/cristoredentorfinal.webp" class="img-fluid"></p>
<p>So, does it look like a painting? Tell us what you think! Try it with your own images and share them with us =). Learning how NST works is a great exercise for improving your understanding of deep learning models in general, especially how we can design cost functions specialized for a given context.</p>
</section>
<section id="learn-deep-learning-with-curso-r" class="level2">
<h2 class="anchored" data-anchor-id="learn-deep-learning-with-curso-r">Learn Deep Learning with Curso-R</h2>
<p>If you want to get into the amazing world of deep networks, our <a href="https://www.curso-r.com/cursos/deep-learning/">Deep Learning</a> course is the best way in. Check it out!</p>



</section>

<div id="quarto-appendix" class="default"><section class="quarto-appendix-contents" id="quarto-bibliography"><h2 class="anchored quarto-appendix-heading">References</h2><div id="refs" class="references csl-bib-body hanging-indent">
<div id="ref-gatys2015neural" class="csl-entry">
Gatys, Leon A., Alexander S. Ecker, and Matthias Bethge. 2015. <em>A Neural Algorithm of Artistic Style</em>. <a href="https://arxiv.org/abs/1508.06576">https://arxiv.org/abs/1508.06576</a>.
</div>
</div></section></div> ]]></description>
  <category>analysis</category>
  <category>concepts</category>
  <category>tutorials</category>
  <category>deep learning</category>
  <category>torch</category>
  <category>neural style transfer</category>
  <category>modeling</category>
  <guid>https://athospd.github.io/athosdamiani/posts/2021-02-22-neural-style-transfer/</guid>
  <pubDate>Mon, 22 Feb 2021 00:00:00 GMT</pubDate>
  <media:content url="https://athospd.github.io/athosdamiani/posts/2021-02-22-neural-style-transfer/cristoredentorfinal.webp" medium="image" type="image/webp"/>
</item>
<item>
  <title>Simple audio classification with torch</title>
  <dc:creator>Athos Damiani</dc:creator>
  <link>https://athospd.github.io/athosdamiani/posts/2021-02-04-simple-audio-classification-with-torch/</link>
  <description><![CDATA[ 





<p><em>First published on the <a href="https://blogs.rstudio.com/ai/posts/2021-02-04-simple-audio-classification-with-torch/">RStudio AI Blog</a> on February 4, 2021.</em></p>
<p>This article translates <a href="https://github.com/dfalbel">Daniel Falbel</a>’s <a href="https://blogs.rstudio.com/ai/posts/2018-06-06-simple-audio-classification-keras/">‘Simple Audio Classification’</a> article from <code>tensorflow/keras</code> to <code>torch/torchaudio</code>. The main goal is to introduce torchaudio and illustrate its contributions to the <code>torch</code> ecosystem. Here, we focus on a popular dataset, the audio loader and the spectrogram transformer. An interesting side product is the parallel between torch and tensorflow, showing sometimes the differences, sometimes the similarities between them.</p>
<!-- https://towardsdatascience.com/a-tale-of-two-frameworks-985fa7fcec#6f89 -->
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb1" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb1-1"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">library</span>(torch)</span>
<span id="cb1-2"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">library</span>(torchaudio)</span></code></pre></div></div>
</div>
<section id="downloading-and-importing" class="level2 page-columns page-full">
<h2 class="anchored" data-anchor-id="downloading-and-importing">Downloading and Importing</h2>
<p>torchaudio has the <code>speechcommand_dataset</code> built in. It filters out background_noise by default and lets us choose between versions <code>v0.01</code> and <code>v0.02</code>.</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb2" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb2-1"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># set an existing folder here to cache the dataset</span></span>
<span id="cb2-2">DATASETS_PATH <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"~/datasets/"</span></span>
<span id="cb2-3"></span>
<span id="cb2-4"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 1.4GB download</span></span>
<span id="cb2-5">df <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">speechcommand_dataset</span>(</span>
<span id="cb2-6">  <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">root =</span> DATASETS_PATH, </span>
<span id="cb2-7">  <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">url =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"speech_commands_v0.01"</span>,</span>
<span id="cb2-8">  <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">download =</span> <span class="cn" style="color: #8f5902;
background-color: null;
font-style: inherit;">TRUE</span></span>
<span id="cb2-9">)</span>
<span id="cb2-10"></span>
<span id="cb2-11"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># expect folder: _background_noise_</span></span>
<span id="cb2-12">df<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span>EXCEPT_FOLDER</span>
<span id="cb2-13"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># [1] "_background_noise_"</span></span>
<span id="cb2-14"></span>
<span id="cb2-15"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># number of audio files</span></span>
<span id="cb2-16"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">length</span>(df)</span>
<span id="cb2-17"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># [1] 64721</span></span>
<span id="cb2-18"></span>
<span id="cb2-19"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># a sample</span></span>
<span id="cb2-20">sample <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> df[<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span>]</span>
<span id="cb2-21"></span>
<span id="cb2-22">sample<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span>waveform[, <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">:</span><span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">10</span>]</span></code></pre></div></div>
</div>
<pre><code>torch_tensor
0.0001 *
 0.9155  0.3052  1.8311  1.8311 -0.3052  0.3052  2.4414  0.9155 -0.9155 -0.6104
[ CPUFloatType{1,10} ]</code></pre>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb4" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb4-1">sample<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span>sample_rate</span>
<span id="cb4-2"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 16000</span></span>
<span id="cb4-3">sample<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span>label</span>
<span id="cb4-4"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># bed</span></span>
<span id="cb4-5"></span>
<span id="cb4-6"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">plot</span>(sample<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span>waveform[<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span>], <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">type =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"l"</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">col =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"royalblue"</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">main =</span> sample<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span>label)</span></code></pre></div></div>
</div>
<div class="cell page-columns page-full">
<div class="cell-output-display page-columns page-full">
<div class="quarto-figure quarto-figure-center page-columns page-full">
<figure class="figure page-columns page-full">
<p class="page-columns page-full"><img src="https://athospd.github.io/athosdamiani/posts/2021-02-04-simple-audio-classification-with-torch/images/bedwave.png" class="img-fluid figure-img column-body-outset" width="516"></p>
<figcaption>A sample waveform for a ‘bed’.</figcaption>
</figure>
</div>
</div>
</div>
</section>
<section id="classes" class="level2">
<h2 class="anchored" data-anchor-id="classes">Classes</h2>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb5" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb5-1">df<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span>classes</span></code></pre></div></div>
</div>
<pre><code> [1] "bed"    "bird"   "cat"    "dog"    "down"   "eight"  "five"  
 [8] "four"   "go"     "happy"  "house"  "left"   "marvin" "nine"  
[15] "no"     "off"    "on"     "one"    "right"  "seven"  "sheila"
[22] "six"    "stop"   "three"  "tree"   "two"    "up"     "wow"   
[29] "yes"    "zero"  </code></pre>
</section>
<section id="generator-dataloader" class="level2">
<h2 class="anchored" data-anchor-id="generator-dataloader"><del>Generator</del> Dataloader</h2>
<p><code>torch::dataloader</code> has the same task as <code>data_generator</code> defined in the original article. It is responsible for preparing batches - including shuffling, padding, one-hot encoding, etc. - and for taking care of parallelism / device I/O orchestration.</p>
<p>In torch we do this by passing the train/test subset to <code>torch::dataloader</code> and encapsulating all the batch setup logic inside a <code>collate_fn()</code> function.</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb7" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb7-1"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">set.seed</span>(<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">6</span>)</span>
<span id="cb7-2">id_train <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">sample</span>(<span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">length</span>(df), <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">size =</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0.7</span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">*</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">length</span>(df))</span>
<span id="cb7-3">id_test <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">setdiff</span>(<span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">seq_len</span>(<span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">length</span>(df)), id_train)</span>
<span id="cb7-4"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># subsets</span></span>
<span id="cb7-5"></span>
<span id="cb7-6">train_subset <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> torch<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">::</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">dataset_subset</span>(df, id_train)</span>
<span id="cb7-7">test_subset <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> torch<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">::</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">dataset_subset</span>(df, id_test)</span></code></pre></div></div>
</div>
<p>At this point, <code>dataloader(train_subset)</code> would not work because the samples are not padded. So we need to build our own <code>collate_fn()</code> with the padding strategy.</p>
<p>I suggest using the following approach when implementing the <code>collate_fn()</code>:</p>
<ol type="1">
<li>begin with <code>collate_fn &lt;- function(batch) browser()</code>.</li>
<li>instantiate <code>dataloader</code> with the <code>collate_fn()</code></li>
<li>create an environment by calling <code>enumerate(dataloader)</code> so you can ask to retrieve a batch from dataloader.</li>
<li>run <code>environment[[1]][[1]]</code>. Now you should be sent inside collate_fn() with access to <code>batch</code> input object.</li>
<li>build the logic.</li>
</ol>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb8" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb8-1">collate_fn <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">function</span>(batch) {</span>
<span id="cb8-2">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">browser</span>()</span>
<span id="cb8-3">}</span>
<span id="cb8-4"></span>
<span id="cb8-5">ds_train <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">dataloader</span>(</span>
<span id="cb8-6">  train_subset, </span>
<span id="cb8-7">  <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">batch_size =</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">32</span>, </span>
<span id="cb8-8">  <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">shuffle =</span> <span class="cn" style="color: #8f5902;
background-color: null;
font-style: inherit;">TRUE</span>, </span>
<span id="cb8-9">  <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">collate_fn =</span> collate_fn</span>
<span id="cb8-10">)</span>
<span id="cb8-11"></span>
<span id="cb8-12">ds_train_env <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">enumerate</span>(ds_train)</span>
<span id="cb8-13">ds_train_env[[<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span>]][[<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span>]]</span></code></pre></div></div>
</div>
<p>The final <code>collate_fn()</code> pads the waveform to length 16001 and then stacks everything up together. At this point there are no spectrograms yet. We going to make spectrogram transformation a part of model architecture.</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb9" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb9-1">pad_sequence <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">function</span>(batch) {</span>
<span id="cb9-2">    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Make all tensors in a batch the same length by padding with zeros</span></span>
<span id="cb9-3">    batch <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">sapply</span>(batch, <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">function</span>(x) (x<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">t</span>()))</span>
<span id="cb9-4">    batch <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> torch<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">::</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">nn_utils_rnn_pad_sequence</span>(batch, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">batch_first =</span> <span class="cn" style="color: #8f5902;
background-color: null;
font-style: inherit;">TRUE</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">padding_value =</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0.</span>)</span>
<span id="cb9-5">    <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">return</span>(batch<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">permute</span>(<span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">c</span>(<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span>, <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">3</span>, <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">2</span>)))</span>
<span id="cb9-6">  }</span>
<span id="cb9-7"></span>
<span id="cb9-8"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Final collate_fn</span></span>
<span id="cb9-9">collate_fn <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">function</span>(batch) {</span>
<span id="cb9-10"> <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Input structure:</span></span>
<span id="cb9-11"> <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># list of 32 lists: list(waveform, sample_rate, label, speaker_id, utterance_number)</span></span>
<span id="cb9-12"> <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Transpose it</span></span>
<span id="cb9-13"> batch <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> purrr<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">::</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">transpose</span>(batch)</span>
<span id="cb9-14"> tensors <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> batch<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span>waveform</span>
<span id="cb9-15"> targets <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> batch<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span>label_index</span>
<span id="cb9-16"></span>
<span id="cb9-17"> <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Group the list of tensors into a batched tensor</span></span>
<span id="cb9-18"> tensors <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">pad_sequence</span>(tensors)</span>
<span id="cb9-19"> </span>
<span id="cb9-20"> <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># target encoding</span></span>
<span id="cb9-21"> targets <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> torch<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">::</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">torch_stack</span>(targets)</span>
<span id="cb9-22"></span>
<span id="cb9-23"> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">list</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">tensors =</span> tensors, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">targets =</span> targets) <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># (64, 1, 16001)</span></span>
<span id="cb9-24">}</span></code></pre></div></div>
</div>
<p>Batch structure is:</p>
<ul>
<li>batch[[1]]: <strong>waveforms</strong> - <code>tensor</code> with dimension (32, 1, 16001)</li>
<li>batch[[2]]: <strong>targets</strong> - <code>tensor</code> with dimension (32, 1)</li>
</ul>
<p>Also, torchaudio comes with 3 loaders, <code>av_loader</code>, <code>tuner_loader</code>, and <code>audiofile_loader</code>- more to come. <code>set_audio_backend()</code> is used to set one of them as the audio loader. Their performances differ based on audio format (mp3 or wav). There is no perfect world yet: <code>tuner_loader</code> is best for mp3, <code>audiofile_loader</code> is best for wav, but neither of them has the option of partially loading a sample from an audio file without bringing all the data into memory first.</p>
<p>For a given audio backend we need pass it to each worker through <code>worker_init_fn()</code> argument.</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb10" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb10-1">ds_train <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">dataloader</span>(</span>
<span id="cb10-2">  train_subset, </span>
<span id="cb10-3">  <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">batch_size =</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">128</span>, </span>
<span id="cb10-4">  <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">shuffle =</span> <span class="cn" style="color: #8f5902;
background-color: null;
font-style: inherit;">TRUE</span>, </span>
<span id="cb10-5">  <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">collate_fn =</span> collate_fn,</span>
<span id="cb10-6">  <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">num_workers =</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">16</span>,</span>
<span id="cb10-7">  <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">worker_init_fn =</span> <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">function</span>(.) {torchaudio<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">::</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">set_audio_backend</span>(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"audiofile_loader"</span>)},</span>
<span id="cb10-8">  <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">worker_globals =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">c</span>(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"pad_sequence"</span>) <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># pad_sequence is needed for collect_fn</span></span>
<span id="cb10-9">)</span>
<span id="cb10-10"></span>
<span id="cb10-11">ds_test <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">dataloader</span>(</span>
<span id="cb10-12">  test_subset, </span>
<span id="cb10-13">  <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">batch_size =</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">64</span>, </span>
<span id="cb10-14">  <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">shuffle =</span> <span class="cn" style="color: #8f5902;
background-color: null;
font-style: inherit;">FALSE</span>, </span>
<span id="cb10-15">  <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">collate_fn =</span> collate_fn,</span>
<span id="cb10-16">  <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">num_workers =</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">8</span>,</span>
<span id="cb10-17">  <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">worker_globals =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">c</span>(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"pad_sequence"</span>) <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># pad_sequence is needed for collect_fn</span></span>
<span id="cb10-18">)</span></code></pre></div></div>
</div>
</section>
<section id="model-definition" class="level2">
<h2 class="anchored" data-anchor-id="model-definition">Model definition</h2>
<p>Instead of <code>keras::keras_model_sequential()</code>, we are going to define a <code>torch::nn_module()</code>. As referenced by the original article, the model is based on <a href="https://keras.rstudio.com/articles/examples/mnist_cnn.html">this architecture for MNIST from this tutorial</a>, and I’ll call it ‘DanielNN’.</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb11" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb11-1">dan_nn <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> torch<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">::</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">nn_module</span>(</span>
<span id="cb11-2">  <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"DanielNN"</span>,</span>
<span id="cb11-3">  </span>
<span id="cb11-4">  <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">initialize =</span> <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">function</span>(</span>
<span id="cb11-5">    <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">window_size_ms =</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">30</span>, </span>
<span id="cb11-6">    <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">window_stride_ms =</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">10</span></span>
<span id="cb11-7">  ) {</span>
<span id="cb11-8">    </span>
<span id="cb11-9">    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># spectrogram spec</span></span>
<span id="cb11-10">    window_size <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">as.integer</span>(<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">16000</span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">*</span>window_size_ms<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">/</span><span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1000</span>)</span>
<span id="cb11-11">    stride <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">as.integer</span>(<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">16000</span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">*</span>window_stride_ms<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">/</span><span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1000</span>)</span>
<span id="cb11-12">    fft_size <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">as.integer</span>(<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">2</span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">^</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">trunc</span>(<span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">log</span>(window_size, <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">2</span>) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span>))</span>
<span id="cb11-13">    n_chunks <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">length</span>(<span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">seq</span>(<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">0</span>, <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">16000</span>, stride))</span>
<span id="cb11-14">    </span>
<span id="cb11-15">    self<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span>spectrogram <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> torchaudio<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">::</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">transform_spectrogram</span>(</span>
<span id="cb11-16">      <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">n_fft =</span> fft_size, </span>
<span id="cb11-17">      <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">win_length =</span> window_size, </span>
<span id="cb11-18">      <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">hop_length =</span> stride, </span>
<span id="cb11-19">      <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">normalized =</span> <span class="cn" style="color: #8f5902;
background-color: null;
font-style: inherit;">TRUE</span>, </span>
<span id="cb11-20">      <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">power =</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">2</span></span>
<span id="cb11-21">    )</span>
<span id="cb11-22">    </span>
<span id="cb11-23">    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># convs 2D</span></span>
<span id="cb11-24">    self<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span>conv1 <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> torch<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">::</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">nn_conv2d</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">in_channels =</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">out_channels =</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">32</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">kernel_size =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">c</span>(<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">3</span>,<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">3</span>))</span>
<span id="cb11-25">    self<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span>conv2 <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> torch<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">::</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">nn_conv2d</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">in_channels =</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">32</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">out_channels =</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">64</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">kernel_size =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">c</span>(<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">3</span>,<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">3</span>))</span>
<span id="cb11-26">    self<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span>conv3 <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> torch<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">::</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">nn_conv2d</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">in_channels =</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">64</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">out_channels =</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">128</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">kernel_size =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">c</span>(<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">3</span>,<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">3</span>))</span>
<span id="cb11-27">    self<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span>conv4 <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> torch<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">::</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">nn_conv2d</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">in_channels =</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">128</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">out_channels =</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">256</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">kernel_size =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">c</span>(<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">3</span>,<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">3</span>))</span>
<span id="cb11-28">    </span>
<span id="cb11-29">    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># denses</span></span>
<span id="cb11-30">    self<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span>dense1 <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> torch<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">::</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">nn_linear</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">in_features =</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">14336</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">out_features =</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">128</span>)</span>
<span id="cb11-31">    self<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span>dense2 <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> torch<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">::</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">nn_linear</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">in_features =</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">128</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">out_features =</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">30</span>)</span>
<span id="cb11-32">  },</span>
<span id="cb11-33">  </span>
<span id="cb11-34">  <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">forward =</span> <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">function</span>(x) {</span>
<span id="cb11-35">    x <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span> <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># (64, 1, 16001)</span></span>
<span id="cb11-36">      self<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">spectrogram</span>() <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span> <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># (64, 1, 257, 101)</span></span>
<span id="cb11-37">      torch<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">::</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">torch_add</span>(<span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0.01</span>) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb11-38">      torch<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">::</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">torch_log</span>() <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb11-39">      self<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">conv1</span>() <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb11-40">      torch<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">::</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">nnf_relu</span>() <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb11-41">      torch<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">::</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">nnf_max_pool2d</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">kernel_size =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">c</span>(<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">2</span>,<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">2</span>)) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb11-42">      </span>
<span id="cb11-43">      self<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">conv2</span>() <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb11-44">      torch<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">::</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">nnf_relu</span>() <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb11-45">      torch<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">::</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">nnf_max_pool2d</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">kernel_size =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">c</span>(<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">2</span>,<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">2</span>)) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb11-46">      </span>
<span id="cb11-47">      self<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">conv3</span>() <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb11-48">      torch<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">::</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">nnf_relu</span>() <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb11-49">      torch<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">::</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">nnf_max_pool2d</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">kernel_size =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">c</span>(<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">2</span>,<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">2</span>)) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb11-50">      </span>
<span id="cb11-51">      self<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">conv4</span>() <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb11-52">      torch<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">::</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">nnf_relu</span>() <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb11-53">      torch<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">::</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">nnf_max_pool2d</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">kernel_size =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">c</span>(<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">2</span>,<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">2</span>)) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb11-54">      </span>
<span id="cb11-55">      torch<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">::</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">nnf_dropout</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">p =</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0.25</span>) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb11-56">      torch<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">::</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">torch_flatten</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">start_dim =</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">2</span>) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb11-57">      </span>
<span id="cb11-58">      self<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">dense1</span>() <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb11-59">      torch<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">::</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">nnf_relu</span>() <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb11-60">      torch<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">::</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">nnf_dropout</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">p =</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0.5</span>) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb11-61">      self<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">dense2</span>() </span>
<span id="cb11-62">  }</span>
<span id="cb11-63">)</span>
<span id="cb11-64"></span>
<span id="cb11-65">model <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">dan_nn</span>()</span>
<span id="cb11-66"></span>
<span id="cb11-67"></span>
<span id="cb11-68">device <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> torch<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">::</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">torch_device</span>(<span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">if</span>(torch<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">::</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">cuda_is_available</span>()) <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"cuda"</span> <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">else</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"cpu"</span>)</span>
<span id="cb11-69">model<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">to</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">device =</span> device)</span>
<span id="cb11-70"></span>
<span id="cb11-71"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">print</span>(model)</span></code></pre></div></div>
</div>
<pre><code>An `nn_module` containing 2,226,846 parameters.

── Modules ──────────────────────────────────────────────────────
● spectrogram: &lt;Spectrogram&gt; #0 parameters
● conv1: &lt;nn_conv2d&gt; #320 parameters
● conv2: &lt;nn_conv2d&gt; #18,496 parameters
● conv3: &lt;nn_conv2d&gt; #73,856 parameters
● conv4: &lt;nn_conv2d&gt; #295,168 parameters
● dense1: &lt;nn_linear&gt; #1,835,136 parameters
● dense2: &lt;nn_linear&gt; #3,870 parameters</code></pre>
</section>
<section id="model-fitting" class="level2">
<h2 class="anchored" data-anchor-id="model-fitting">Model fitting</h2>
<p>Unlike in tensorflow, there is no <code>model %&gt;% compile(...)</code> step in torch, so we are going to set <code>loss criterion</code>, <code>optimizer strategy</code> and <code>evaluation metrics</code> explicitly in the training loop.</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb13" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb13-1">loss_criterion <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> torch<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">::</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">nn_cross_entropy_loss</span>()</span>
<span id="cb13-2">optimizer <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> torch<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">::</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">optim_adadelta</span>(model<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span>parameters, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">rho =</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0.95</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">eps =</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">1e-7</span>)</span>
<span id="cb13-3">metrics <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">list</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">acc =</span> yardstick<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">::</span>accuracy_vec)</span></code></pre></div></div>
</div>
<section id="training-loop" class="level3">
<h3 class="anchored" data-anchor-id="training-loop">Training loop</h3>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb14" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb14-1"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">library</span>(glue)</span>
<span id="cb14-2"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">library</span>(progress)</span>
<span id="cb14-3"></span>
<span id="cb14-4">pred_to_r <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">function</span>(x) {</span>
<span id="cb14-5">  classes <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">factor</span>(df<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span>classes)</span>
<span id="cb14-6">  classes[<span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">as.numeric</span>(x<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">to</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">device =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"cpu"</span>))]</span>
<span id="cb14-7">}</span>
<span id="cb14-8"></span>
<span id="cb14-9">set_progress_bar <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">function</span>(total) {</span>
<span id="cb14-10">  progress_bar<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">new</span>(</span>
<span id="cb14-11">    <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">total =</span> total, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">clear =</span> <span class="cn" style="color: #8f5902;
background-color: null;
font-style: inherit;">FALSE</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">width =</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">70</span>,</span>
<span id="cb14-12">    <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">format =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">":current/:total [:bar] - :elapsed - loss: :loss - acc: :acc"</span></span>
<span id="cb14-13">  )</span>
<span id="cb14-14">}</span></code></pre></div></div>
</div>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb15" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb15-1">epochs <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">20</span></span>
<span id="cb15-2">losses <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">c</span>()</span>
<span id="cb15-3">accs <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">c</span>()</span>
<span id="cb15-4"></span>
<span id="cb15-5"><span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">for</span>(epoch <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">in</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">seq_len</span>(epochs)) {</span>
<span id="cb15-6">  pb <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">set_progress_bar</span>(<span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">length</span>(ds_train))</span>
<span id="cb15-7">  pb<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">message</span>(<span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">glue</span>(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Epoch {epoch}/{epochs}"</span>))</span>
<span id="cb15-8">  coro<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">::</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">loop</span>(<span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">for</span>(batch <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">in</span> ds_train) {</span>
<span id="cb15-9">    optimizer<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">zero_grad</span>()</span>
<span id="cb15-10">    predictions <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">model</span>(batch[[<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span>]]<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">to</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">device =</span> device))</span>
<span id="cb15-11">    targets <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> batch[[<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">2</span>]]<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">to</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">device =</span> device)</span>
<span id="cb15-12">    loss <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">loss_criterion</span>(predictions, targets)</span>
<span id="cb15-13">    loss<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">backward</span>()</span>
<span id="cb15-14">    optimizer<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">step</span>()</span>
<span id="cb15-15">    </span>
<span id="cb15-16">    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># eval reports</span></span>
<span id="cb15-17">    prediction_r <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">pred_to_r</span>(predictions<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">argmax</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">dim =</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">2</span>))</span>
<span id="cb15-18">    targets_r <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">pred_to_r</span>(targets)</span>
<span id="cb15-19">    acc <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> metrics<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">acc</span>(targets_r, prediction_r)</span>
<span id="cb15-20">    accs <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">c</span>(accs, acc)</span>
<span id="cb15-21">    loss_r <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">as.numeric</span>(loss<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">item</span>())</span>
<span id="cb15-22">    losses <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">c</span>(losses, loss_r)</span>
<span id="cb15-23">    </span>
<span id="cb15-24">    pb<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">tick</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">tokens =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">list</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">loss =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">round</span>(<span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">mean</span>(losses), <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">4</span>), <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">acc =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">round</span>(<span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">mean</span>(accs), <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">4</span>)))</span>
<span id="cb15-25">  })</span>
<span id="cb15-26">}</span>
<span id="cb15-27"></span>
<span id="cb15-28"></span>
<span id="cb15-29"></span>
<span id="cb15-30"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># test</span></span>
<span id="cb15-31">predictions_r <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">c</span>()</span>
<span id="cb15-32">targets_r <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">c</span>()</span>
<span id="cb15-33">coro<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">::</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">loop</span>(<span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">for</span>(batch_test <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">in</span> ds_test) {</span>
<span id="cb15-34">  predictions <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">model</span>(batch_test[[<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span>]]<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">to</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">device =</span> device))</span>
<span id="cb15-35">  targets <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> batch_test[[<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">2</span>]]<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">to</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">device =</span> device)</span>
<span id="cb15-36">  predictions_r <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">c</span>(predictions_r, <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">pred_to_r</span>(predictions<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">argmax</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">dim =</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">2</span>)))</span>
<span id="cb15-37">  targets_r <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">c</span>(targets_r, <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">pred_to_r</span>(targets))</span>
<span id="cb15-38">})</span>
<span id="cb15-39">val_acc <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> metrics<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">acc</span>(<span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">factor</span>(targets_r, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">levels =</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">:</span><span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">30</span>), <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">factor</span>(predictions_r, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">levels =</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">:</span><span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">30</span>))</span>
<span id="cb15-40"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">cat</span>(<span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">glue</span>(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"val_acc: {val_acc}</span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">\n\n</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"</span>))</span></code></pre></div></div>
</div>
<pre><code>Epoch 1/20                                                            
[W SpectralOps.cpp:590] Warning: The function torch.rfft is deprecated and will be removed in a future PyTorch release. Use the new torch.fft module functions, instead, by importing torch.fft and calling torch.fft.fft or torch.fft.rfft. (function operator())
354/354 [=========================] -  1m - loss: 2.6102 - acc: 0.2333
Epoch 2/20                                                            
354/354 [=========================] -  1m - loss: 1.9779 - acc: 0.4138
Epoch 3/20                                                            
354/354 [============================] -  1m - loss: 1.62 - acc: 0.519
Epoch 4/20                                                            
354/354 [=========================] -  1m - loss: 1.3926 - acc: 0.5859
Epoch 5/20                                                            
354/354 [==========================] -  1m - loss: 1.2334 - acc: 0.633
Epoch 6/20                                                            
354/354 [=========================] -  1m - loss: 1.1135 - acc: 0.6685
Epoch 7/20                                                            
354/354 [=========================] -  1m - loss: 1.0199 - acc: 0.6961
Epoch 8/20                                                            
354/354 [=========================] -  1m - loss: 0.9444 - acc: 0.7181
Epoch 9/20                                                            
354/354 [=========================] -  1m - loss: 0.8816 - acc: 0.7365
Epoch 10/20                                                           
354/354 [=========================] -  1m - loss: 0.8278 - acc: 0.7524
Epoch 11/20                                                           
354/354 [=========================] -  1m - loss: 0.7818 - acc: 0.7659
Epoch 12/20                                                           
354/354 [=========================] -  1m - loss: 0.7413 - acc: 0.7778
Epoch 13/20                                                           
354/354 [=========================] -  1m - loss: 0.7064 - acc: 0.7881
Epoch 14/20                                                           
354/354 [=========================] -  1m - loss: 0.6751 - acc: 0.7974
Epoch 15/20                                                           
354/354 [=========================] -  1m - loss: 0.6469 - acc: 0.8058
Epoch 16/20                                                           
354/354 [=========================] -  1m - loss: 0.6216 - acc: 0.8133
Epoch 17/20                                                           
354/354 [=========================] -  1m - loss: 0.5985 - acc: 0.8202
Epoch 18/20                                                           
354/354 [=========================] -  1m - loss: 0.5774 - acc: 0.8263
Epoch 19/20                                                           
354/354 [==========================] -  1m - loss: 0.5582 - acc: 0.832
Epoch 20/20                                                           
354/354 [=========================] -  1m - loss: 0.5403 - acc: 0.8374
val_acc: 0.876705979296493</code></pre>
</section>
</section>
<section id="making-predictions" class="level2 page-columns page-full">
<h2 class="anchored" data-anchor-id="making-predictions">Making predictions</h2>
<p>We already have all predictions calculated for <code>test_subset</code>, let’s recreate the alluvial plot from the original article.</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb17" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb17-1"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">library</span>(dplyr)</span>
<span id="cb17-2"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">library</span>(alluvial)</span>
<span id="cb17-3">df_validation <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">data.frame</span>(</span>
<span id="cb17-4">  <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">pred_class =</span> df<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span>classes[predictions_r],</span>
<span id="cb17-5">  <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">class =</span> df<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span>classes[targets_r]</span>
<span id="cb17-6">)</span>
<span id="cb17-7">x <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span>  df_validation <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb17-8">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">mutate</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">correct =</span> pred_class <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">==</span> class) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb17-9">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">count</span>(pred_class, class, correct)</span>
<span id="cb17-10"></span>
<span id="cb17-11"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">alluvial</span>(</span>
<span id="cb17-12">  x <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">select</span>(class, pred_class),</span>
<span id="cb17-13">  <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">freq =</span> x<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span>n,</span>
<span id="cb17-14">  <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">col =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">ifelse</span>(x<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span>correct, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"lightblue"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"red"</span>),</span>
<span id="cb17-15">  <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">border =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">ifelse</span>(x<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span>correct, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"lightblue"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"red"</span>),</span>
<span id="cb17-16">  <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">alpha =</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0.6</span>,</span>
<span id="cb17-17">  <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">hide =</span> x<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span>n <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">20</span></span>
<span id="cb17-18">)</span></code></pre></div></div>
</div>
<div class="cell page-columns page-full">
<div class="cell-output-display page-columns page-full">
<div class="quarto-figure quarto-figure-center page-columns page-full">
<figure class="figure page-columns page-full">
<p class="page-columns page-full"><img src="https://athospd.github.io/athosdamiani/posts/2021-02-04-simple-audio-classification-with-torch/images/alluvial.png" class="img-fluid figure-img column-body-outset" width="336"></p>
<figcaption>Model performance: true labels &lt;–&gt; predicted labels.</figcaption>
</figure>
</div>
</div>
</div>
<p>Model accuracy is 87,7%, somewhat worse than tensorflow version from the original post. Nevertheless, all conclusions from original post still hold.</p>


</section>

 ]]></description>
  <category>torch</category>
  <category>r</category>
  <category>audio processing</category>
  <guid>https://athospd.github.io/athosdamiani/posts/2021-02-04-simple-audio-classification-with-torch/</guid>
  <pubDate>Thu, 04 Feb 2021 00:00:00 GMT</pubDate>
  <media:content url="https://athospd.github.io/athosdamiani/posts/2021-02-04-simple-audio-classification-with-torch/images/preview.jpg" medium="image" type="image/jpeg" height="96" width="144"/>
</item>
<item>
  <title>{tabulizer}: PDF table to data.frame in seconds</title>
  <dc:creator>Athos Damiani</dc:creator>
  <link>https://athospd.github.io/athosdamiani/posts/2021-01-08-tabulizer/</link>
  <description><![CDATA[ 





<section id="summary" class="level2">
<h2 class="anchored" data-anchor-id="summary">Summary</h2>
<blockquote class="blockquote">
<p><code>tabulizer::extract_tables("caminho/do/arquivo.pdf", pages = 153)</code></p>
</blockquote>
<p>That’s all!! I decided to write this post because I was happy with how easy it was to bring a table from a PDF into R after CTLR+C/CTRL+V failed miserably.</p>
<p>A potential obstacle may be the infamous {rJava}, which {tabulizer} depends on. But on <strong>linux</strong> I ran <code>sudo apt install default-jdk default-jre</code> and it worked for me! I haven’t tested it on <strong>windows</strong>, but there’s <a href="https://mauriciogeneroso.medium.com/configurando-java-1-instala%C3%A7%C3%A3o-do-jre-e-do-jdk-no-windows-38cacace0377">this tutorial here</a> (in Portuguese).</p>
</section>
<section id="task" class="level2">
<h2 class="anchored" data-anchor-id="task">Task</h2>
<p>Extract a table from a page of a pdf file. <code>url_pdf</code> holds the address of a PDF on the internet.</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb1" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb1-1">url_pdf <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"https://curso-r.github.io/main-regressao-linear/referencias/Ci%C3%AAncia%20de%20Dados.%20Fundamentos%20e%20Aplica%C3%A7%C3%B5es.%20Vers%C3%A3o%20parcial%20preliminar.%20maio%20Pedro%20A.%20Morettin%20Julio%20M.%20Singer.pdf"</span></span></code></pre></div></div>
</div>
</section>
<section id="a-look-at-the-pdf-page" class="level2">
<h2 class="anchored" data-anchor-id="a-look-at-the-pdf-page">A look at the PDF page</h2>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb2" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb2-1"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># takes a screenshot of page 153 and saves it as a PNG image.</span></span>
<span id="cb2-2">pdftools<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">::</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">pdf_convert</span>(url_pdf, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">pages =</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">153</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">filenames =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"pag153.png"</span>)</span>
<span id="cb2-3"><span class="do" style="color: #5E5E5E;
background-color: null;
font-style: italic;">## Converting page 153 to pag153.png... done!</span></span>
<span id="cb2-4"><span class="do" style="color: #5E5E5E;
background-color: null;
font-style: italic;">## [1] "pag153.png"</span></span></code></pre></div></div>
</div>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb3" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb3-1"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># shows the PNG image.</span></span>
<span id="cb3-2">knitr<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">::</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">include_graphics</span>(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"pag153.webp"</span>)</span></code></pre></div></div>
<div class="cell-output-display">
<div>
<figure class="figure">
<p><img src="https://athospd.github.io/athosdamiani/posts/2021-01-08-tabulizer/pag153.webp" class="img-fluid figure-img"></p>
</figure>
</div>
</div>
</div>
</section>
<section id="extracting-the-table-pdf---r" class="level2">
<h2 class="anchored" data-anchor-id="extracting-the-table-pdf---r">Extracting the table PDF -&gt; R</h2>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb4" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb4-1"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># extracts the table from the PDF (not from the PNG!)</span></span>
<span id="cb4-2">tabela_extrida_do_pdf <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> tabulizer<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">::</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">extract_tables</span>(url_pdf, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">pages =</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">153</span>)</span>
<span id="cb4-3">tabela_extrida_do_pdf[[<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span>]]</span>
<span id="cb4-4"><span class="do" style="color: #5E5E5E;
background-color: null;
font-style: italic;">##       [,1]    [,2]     [,3]        [,4]    [,5]     [,6]       </span></span>
<span id="cb4-5"><span class="do" style="color: #5E5E5E;
background-color: null;
font-style: italic;">##  [1,] ""      "Idade"  "Distância" ""      "Idade"  "Distância"</span></span>
<span id="cb4-6"><span class="do" style="color: #5E5E5E;
background-color: null;
font-style: italic;">##  [2,] "Ident" "(anos)" "(m)"       "Ident" "(anos)" "(m)"      </span></span>
<span id="cb4-7"><span class="do" style="color: #5E5E5E;
background-color: null;
font-style: italic;">##  [3,] "1"     "18"     "170"       "16"    "55"     "140"      </span></span>
<span id="cb4-8"><span class="do" style="color: #5E5E5E;
background-color: null;
font-style: italic;">##  [4,] "2"     "20"     "197"       "17"    "63"     "117"      </span></span>
<span id="cb4-9"><span class="do" style="color: #5E5E5E;
background-color: null;
font-style: italic;">##  [5,] "3"     "22"     "187"       "18"    "65"     "140"      </span></span>
<span id="cb4-10"><span class="do" style="color: #5E5E5E;
background-color: null;
font-style: italic;">##  [6,] "4"     "23"     "170"       "19"    "66"     "100"      </span></span>
<span id="cb4-11"><span class="do" style="color: #5E5E5E;
background-color: null;
font-style: italic;">##  [7,] "5"     "23"     "153"       "20"    "67"     "137"      </span></span>
<span id="cb4-12"><span class="do" style="color: #5E5E5E;
background-color: null;
font-style: italic;">##  [8,] "6"     "25"     "163"       "21"    "68"     "100"      </span></span>
<span id="cb4-13"><span class="do" style="color: #5E5E5E;
background-color: null;
font-style: italic;">##  [9,] "7"     "27"     "187"       "22"    "70"     "130"      </span></span>
<span id="cb4-14"><span class="do" style="color: #5E5E5E;
background-color: null;
font-style: italic;">## [10,] "8"     "28"     "170"       "23"    "71"     "107"      </span></span>
<span id="cb4-15"><span class="do" style="color: #5E5E5E;
background-color: null;
font-style: italic;">## [11,] "9"     "29"     "153"       "24"    "72"     "123"      </span></span>
<span id="cb4-16"><span class="do" style="color: #5E5E5E;
background-color: null;
font-style: italic;">## [12,] "10"    "32"     "137"       "25"    "73"     "93"       </span></span>
<span id="cb4-17"><span class="do" style="color: #5E5E5E;
background-color: null;
font-style: italic;">## [13,] "11"    "37"     "140"       "26"    "74"     "140"      </span></span>
<span id="cb4-18"><span class="do" style="color: #5E5E5E;
background-color: null;
font-style: italic;">## [14,] "12"    "41"     "153"       "27"    "75"     "153"      </span></span>
<span id="cb4-19"><span class="do" style="color: #5E5E5E;
background-color: null;
font-style: italic;">## [15,] "13"    "46"     "150"       "28"    "77"     "120"      </span></span>
<span id="cb4-20"><span class="do" style="color: #5E5E5E;
background-color: null;
font-style: italic;">## [16,] "14"    "49"     "127"       "29"    "79"     "103"      </span></span>
<span id="cb4-21"><span class="do" style="color: #5E5E5E;
background-color: null;
font-style: italic;">## [17,] "15"    "53"     "153"       "30"    "82"     "120"</span></span></code></pre></div></div>
</div>
<p>QED! ⬛</p>
</section>
<section id="bonus-cleaning-up-the-table" class="level2">
<h2 class="anchored" data-anchor-id="bonus-cleaning-up-the-table">Bonus: Cleaning up the table</h2>
<p>Once it’s inside R, it’s game on. Let’s clean it up, which is what we do best. The challenge is to stack the columns in groups of 3, <code>Ident, (anos), (m)</code>. We have two individuals per row, and we need to stack them so that in the end we have one row per individual, by the book. The hero here is <code>pivot_longer()</code>.</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb5" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb5-1"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">library</span>(tidyverse)</span>
<span id="cb5-2"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">library</span>(janitor)</span>
<span id="cb5-3"></span>
<span id="cb5-4">tabela_extrida_do_pdf[[<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span>]] <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb5-5">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">as_tibble</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">.name_repair =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"unique"</span>) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb5-6">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">row_to_names</span>(<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">2</span>) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb5-7">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">clean_names</span>() <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb5-8">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">pivot_longer</span>(<span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">everything</span>(), <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">names_to =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">c</span>(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">".value"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"conjunto"</span>), <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">names_sep =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"_"</span>) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb5-9">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">select</span>(<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">-</span>conjunto) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb5-10">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">head</span>() <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb5-11">  kableExtra<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">::</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">kable</span>()</span></code></pre></div></div>
<div class="cell-output-display">
<table class="caption-top table table-sm table-striped small">
<thead>
<tr class="header">
<th style="text-align: left;">ident</th>
<th style="text-align: left;">anos</th>
<th style="text-align: left;">m</th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td style="text-align: left;">1</td>
<td style="text-align: left;">18</td>
<td style="text-align: left;">170</td>
</tr>
<tr class="even">
<td style="text-align: left;">16</td>
<td style="text-align: left;">55</td>
<td style="text-align: left;">140</td>
</tr>
<tr class="odd">
<td style="text-align: left;">2</td>
<td style="text-align: left;">20</td>
<td style="text-align: left;">197</td>
</tr>
<tr class="even">
<td style="text-align: left;">17</td>
<td style="text-align: left;">63</td>
<td style="text-align: left;">117</td>
</tr>
<tr class="odd">
<td style="text-align: left;">3</td>
<td style="text-align: left;">22</td>
<td style="text-align: left;">187</td>
</tr>
<tr class="even">
<td style="text-align: left;">18</td>
<td style="text-align: left;">65</td>
<td style="text-align: left;">140</td>
</tr>
</tbody>
</table>
</div>
</div>
</section>
<section id="related-reading" class="level2">
<h2 class="anchored" data-anchor-id="related-reading">Related reading</h2>
<p>I suggest reading the post <a href="https://www.curso-r.com/blog/2017-05-27-ocr/">PDF e OCR</a> (in Portuguese) by <a href="">Julio Trecenti</a> for more advanced features of {pdftools}, {tesseract} and {tabulizer}.</p>
</section>
<section id="random-comment" class="level2">
<h2 class="anchored" data-anchor-id="random-comment">Random comment</h2>
<p>Take the chance to check out the 2020 book <a href="https://docplayer.com.br/156868089-Ciencia-de-dados-fundamentos-e-aplicacoes-versao-parcial-preliminar-maio-pedro-a-morettin-julio-m-singer.html"><strong>Ciência de dados: Fundamentos e Prática</strong></a> (in Portuguese) by Professors Julio Singer and Renato Morettin.</p>
<p>Thanks!</p>


</section>

 ]]></description>
  <category>tutorials</category>
  <category>data cleaning</category>
  <category>pdf</category>
  <category>tabulizer</category>
  <category>pdftools</category>
  <guid>https://athospd.github.io/athosdamiani/posts/2021-01-08-tabulizer/</guid>
  <pubDate>Fri, 08 Jan 2021 00:00:00 GMT</pubDate>
  <media:content url="https://athospd.github.io/athosdamiani/posts/2021-01-08-tabulizer/tabulizer.webp" medium="image" type="image/webp"/>
</item>
<item>
  <title>XGBoost predictions straight from SQL</title>
  <dc:creator>Athos Damiani</dc:creator>
  <link>https://athospd.github.io/athosdamiani/posts/2020-08-29-xgboost-sql/</link>
  <description><![CDATA[ 





<p>What you’ll learn to do by the end of this post is:</p>
<p><img src="https://athospd.github.io/athosdamiani/posts/2020-08-29-xgboost-sql/xgboost_para_sql.gif"></p>
<p>That is, once you have an XGBoost model fitted in your R, you can ask {tidypredict} to generate the SQL query that would compute the score. PS: the XGBoost in the gif has 2 trees.</p>
<section id="reasons-to-use-it" class="level3">
<h3 class="anchored" data-anchor-id="reasons-to-use-it">Reasons to use it</h3>
<p>If there is a very powerful SQL server (or cluster, etc), the prediction can be done directly inside the database, with no need for other tools. Also, putting it into production will be a CTRL+C/CTRL+V away =P.</p>
</section>
<section id="the-xgboost-predictor" class="level3">
<h3 class="anchored" data-anchor-id="the-xgboost-predictor">The XGBoost predictor</h3>
<p>The value predicted by XGBoost is, deep down, a sum of CASE WHEN’s (or IF ELSE’s). The number of trees defines how many CASE WHEN’s will be added up.</p>
<p>The predictor of an XGBoost with 2 trees would look like this in SQL:</p>
<pre><code>&lt;SQL&gt; 0.0 + CASE
WHEN ((`dose` &lt; 5.0 OR ((`dose`) IS NULL))) THEN (-1.95000005)
WHEN (`dose` &gt;= 14.0 AND `dose` &gt;= 5.0) THEN (-1.6500001)
WHEN ((`dose` &lt; 10.0 OR ((`dose`) IS NULL)) AND (`dose` &lt; 14.0 OR ((`dose`) IS NULL)) AND `dose` &gt;= 5.0) THEN (1.05000007)
WHEN (`dose` &gt;= 10.0 AND (`dose` &lt; 14.0 OR ((`dose`) IS NULL)) AND `dose` &gt;= 5.0) THEN (1.35000002)
END + CASE
WHEN ((`dose` &lt; 5.0 OR ((`dose`) IS NULL))) THEN (-1.36500013)
WHEN (`dose` &gt;= 14.0 AND `dose` &gt;= 5.0) THEN (-1.15499997)
WHEN ((`dose` &lt; 10.0 OR ((`dose`) IS NULL)) AND (`dose` &lt; 14.0 OR ((`dose`) IS NULL)) AND `dose` &gt;= 5.0) THEN (0.734999955)
WHEN (`dose` &gt;= 10.0 AND (`dose` &lt; 14.0 OR ((`dose`) IS NULL)) AND `dose` &gt;= 5.0) THEN (0.945000052)
END + 0.5</code></pre>
</section>
<section id="minimal-code" class="level3">
<h3 class="anchored" data-anchor-id="minimal-code">Minimal code</h3>
<p>The true hero of the code below is the line <code>tidypredict_sql(xgb_fit$fit, con)</code> from the <a href="https://tidypredict.tidymodels.org/"><code>{tidypredict}</code></a> package, which takes the fitted XGBoost object and returns the SQL query with the CASE WHEN’s.</p>
<section id="fitting-the-xgboost-with-tidymodels" class="level4">
<h4 class="anchored" data-anchor-id="fitting-the-xgboost-with-tidymodels">Fitting the XGBoost with {tidymodels}</h4>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb2" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb2-1"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">library</span>(tidymodels)</span>
<span id="cb2-2"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">library</span>(tidypredict)</span>
<span id="cb2-3"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">library</span>(xgboost)</span>
<span id="cb2-4"></span>
<span id="cb2-5"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># data -----------------------------------------------------</span></span>
<span id="cb2-6">data <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">tibble</span>(</span>
<span id="cb2-7">  <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">dose =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">c</span>(<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">2</span>, <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">8</span>, <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">12</span>, <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">18</span>),</span>
<span id="cb2-8">  <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">eficacia =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">c</span>(<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">-</span><span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">6</span>, <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">4</span>, <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">5</span>, <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">-</span><span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">5</span>)</span>
<span id="cb2-9">)</span>
<span id="cb2-10"></span>
<span id="cb2-11"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># model specification -------------------------------------</span></span>
<span id="cb2-12">xgb_model <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">boost_tree</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">min_n =</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span>, </span>
<span id="cb2-13">  <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">trees =</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">2</span></span>
<span id="cb2-14">) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb2-15">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">set_engine</span>(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"xgboost"</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">lambda =</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">0</span>) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb2-16">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">set_mode</span>(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"regression"</span>)</span>
<span id="cb2-17"></span>
<span id="cb2-18"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># model fit ----------------------------------------------</span></span>
<span id="cb2-19">xgb_fit <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">fit</span>(xgb_model, eficacia <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">~</span> dose, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">data =</span> data)</span>
<span id="cb2-20"></span>
<span id="cb2-21"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Connection to the SQL database --------------------------</span></span>
<span id="cb2-22">con <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> DBI<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">::</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">dbConnect</span>(RSQLite<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">::</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">SQLite</span>(), <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"meu_sqlite_db.db"</span>)</span>
<span id="cb2-23"></span>
<span id="cb2-24"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Transcribing XGBoost to SQL -----------------------------</span></span>
<span id="cb2-25"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">tidypredict_sql</span>(xgb_fit<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span>fit, con)</span>
<span id="cb2-26"><span class="do" style="color: #5E5E5E;
background-color: null;
font-style: italic;">## &lt;SQL&gt; (CASE</span></span>
<span id="cb2-27"><span class="do" style="color: #5E5E5E;
background-color: null;
font-style: italic;">## WHEN (`dose` &lt; 15.0 OR (`dose` IS NULL)) THEN -1.6500001</span></span>
<span id="cb2-28"><span class="do" style="color: #5E5E5E;
background-color: null;
font-style: italic;">## ELSE -1.95000005</span></span>
<span id="cb2-29"><span class="do" style="color: #5E5E5E;
background-color: null;
font-style: italic;">## </span><span class="re">END</span><span class="do" style="color: #5E5E5E;
background-color: null;
font-style: italic;"> + CASE</span></span>
<span id="cb2-30"><span class="do" style="color: #5E5E5E;
background-color: null;
font-style: italic;">## WHEN (`dose` &lt; 10.0 OR (`dose` IS NULL)) THEN 0.734999955</span></span>
<span id="cb2-31"><span class="do" style="color: #5E5E5E;
background-color: null;
font-style: italic;">## ELSE 0.945000052</span></span>
<span id="cb2-32"><span class="do" style="color: #5E5E5E;
background-color: null;
font-style: italic;">## </span><span class="re">END</span><span class="do" style="color: #5E5E5E;
background-color: null;
font-style: italic;">) + 0.5</span></span></code></pre></div></div>
</div>
</section>
<section id="using-dplyr-to-run-code-in-sql" class="level4">
<h4 class="anchored" data-anchor-id="using-dplyr-to-run-code-in-sql">Using {dplyr} to run code in SQL</h4>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb3" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb3-1"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># just uploading the data to the SQL database --------</span></span>
<span id="cb3-2"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">copy_to</span>(con, data, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"data"</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">overwrite =</span> <span class="cn" style="color: #8f5902;
background-color: null;
font-style: inherit;">TRUE</span>)</span>
<span id="cb3-3"></span>
<span id="cb3-4"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Creating the `pred` column with the SQL predictions</span></span>
<span id="cb3-5">data_sql <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">tbl</span>(con, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"data"</span>) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb3-6">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">mutate</span>(</span>
<span id="cb3-7">    <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">pred =</span> <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">!!</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">tidypredict_sql</span>(xgb_fit<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span>fit, con)</span>
<span id="cb3-8">  )</span>
<span id="cb3-9"></span>
<span id="cb3-10"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># result ----------------------------------------------</span></span>
<span id="cb3-11">data_sql</span>
<span id="cb3-12"><span class="do" style="color: #5E5E5E;
background-color: null;
font-style: italic;">## # Source:   SQL [4 x 3]</span></span>
<span id="cb3-13"><span class="do" style="color: #5E5E5E;
background-color: null;
font-style: italic;">## # Database: sqlite 3.51.2 [C:\Users\ap_da\athosdamiani\posts\2020-08-29-xgboost-sql\meu_sqlite_db.db]</span></span>
<span id="cb3-14"><span class="do" style="color: #5E5E5E;
background-color: null;
font-style: italic;">##    dose eficacia   pred</span></span>
<span id="cb3-15"><span class="do" style="color: #5E5E5E;
background-color: null;
font-style: italic;">##   &lt;dbl&gt;    &lt;dbl&gt;  &lt;dbl&gt;</span></span>
<span id="cb3-16"><span class="do" style="color: #5E5E5E;
background-color: null;
font-style: italic;">## 1     2       -6 -0.415</span></span>
<span id="cb3-17"><span class="do" style="color: #5E5E5E;
background-color: null;
font-style: italic;">## 2     8        4 -0.415</span></span>
<span id="cb3-18"><span class="do" style="color: #5E5E5E;
background-color: null;
font-style: italic;">## 3    12        5 -0.205</span></span>
<span id="cb3-19"><span class="do" style="color: #5E5E5E;
background-color: null;
font-style: italic;">## 4    18       -5 -0.505</span></span></code></pre></div></div>
</div>
<p>PS: the <code>tidypredict_sql()</code> function returns a string, so we use the <code>!!</code> operator to interpret it as literal R code. This part can be confusing because dplyr will take the code and turn it into the literal, but it <strong>won’t</strong> run the code in R, it will translate from R to SQL directly (and that’s why it works, because the string isn’t valid R code =P).</p>
</section>
<section id="sql-behind-the-scenes" class="level4">
<h4 class="anchored" data-anchor-id="sql-behind-the-scenes">SQL behind the scenes</h4>
<p>The <code>data_sql</code> table generated above by {dplyr} is actually an SQL query. The <code>show_query()</code> function shows the resulting translation:</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb4" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb4-1"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">show_query</span>(data_sql)</span>
<span id="cb4-2"><span class="do" style="color: #5E5E5E;
background-color: null;
font-style: italic;">## &lt;SQL&gt;</span></span>
<span id="cb4-3"><span class="do" style="color: #5E5E5E;
background-color: null;
font-style: italic;">## SELECT</span></span>
<span id="cb4-4"><span class="do" style="color: #5E5E5E;
background-color: null;
font-style: italic;">##   `data`.*,</span></span>
<span id="cb4-5"><span class="do" style="color: #5E5E5E;
background-color: null;
font-style: italic;">##   (CASE</span></span>
<span id="cb4-6"><span class="do" style="color: #5E5E5E;
background-color: null;
font-style: italic;">## WHEN (`dose` &lt; 15.0 OR (`dose` IS NULL)) THEN -1.6500001</span></span>
<span id="cb4-7"><span class="do" style="color: #5E5E5E;
background-color: null;
font-style: italic;">## ELSE -1.95000005</span></span>
<span id="cb4-8"><span class="do" style="color: #5E5E5E;
background-color: null;
font-style: italic;">## </span><span class="re">END</span><span class="do" style="color: #5E5E5E;
background-color: null;
font-style: italic;"> + CASE</span></span>
<span id="cb4-9"><span class="do" style="color: #5E5E5E;
background-color: null;
font-style: italic;">## WHEN (`dose` &lt; 10.0 OR (`dose` IS NULL)) THEN 0.734999955</span></span>
<span id="cb4-10"><span class="do" style="color: #5E5E5E;
background-color: null;
font-style: italic;">## ELSE 0.945000052</span></span>
<span id="cb4-11"><span class="do" style="color: #5E5E5E;
background-color: null;
font-style: italic;">## </span><span class="re">END</span><span class="do" style="color: #5E5E5E;
background-color: null;
font-style: italic;">) + 0.5 AS `pred`</span></span>
<span id="cb4-12"><span class="do" style="color: #5E5E5E;
background-color: null;
font-style: italic;">## FROM `data`</span></span></code></pre></div></div>
</div>
</section>
</section>
<section id="summary" class="level2">
<h2 class="anchored" data-anchor-id="summary">Summary</h2>
<ul>
<li>XGBoost to SQL is as simple as one line of code: <code>tidypredict_sql(xgb_fit$fit, con)</code></li>
<li>The SQL query can be used inside <code>mutate()</code> with <code>!!</code></li>
<li>XGBoost predictions are no rocket science, they are simple sums of CASE WHEN’s.</li>
</ul>
<section id="appendix-the-two-trees" class="level3">
<h3 class="anchored" data-anchor-id="appendix-the-two-trees">Appendix: the two trees</h3>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb5" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb5-1"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">library</span>(DiagrammeR)</span>
<span id="cb5-2"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">xgb.plot.tree</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">model=</span>xgb_fit<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span>fit)</span></code></pre></div></div>
<div class="cell-output-display">
<div class="grViz html-widget html-fill-item" id="htmlwidget-585e52f47b3a081e851e" style="width:100%;height:464px;"></div>
<script type="application/json" data-for="htmlwidget-585e52f47b3a081e851e">{"x":{"diagram":"digraph {\n\ngraph [layout = \"dot\",\n       rankdir = \"LR\"]\n\nnode [color = \"DimGray\",\n      style = \"filled\",\n      fontname = \"Helvetica\"]\n\nedge [color = \"DimGray\",\n     arrowsize = \"1.5\",\n     arrowhead = \"vee\",\n     fontname = \"Helvetica\"]\n\n  \"1\" [label = \"Tree 1\ndose\nCover: 4\nGain: 19.7633343\", fillcolor = \"Beige\", shape = \"rectangle\", fontcolor = \"black\"] \n  \"2\" [label = \"Leaf\nCover: 1\nValue: -1.36500013\", fillcolor = \"Khaki\", shape = \"oval\", fontcolor = \"black\"] \n  \"3\" [label = \"dose\nCover: 3\nGain: 29.4816647\", fillcolor = \"Beige\", shape = \"rectangle\", fontcolor = \"black\"] \n  \"4\" [label = \"dose\nCover: 2\nGain: 0.244999886\", fillcolor = \"Beige\", shape = \"rectangle\", fontcolor = \"black\"] \n  \"5\" [label = \"Leaf\nCover: 1\nValue: -1.15499997\", fillcolor = \"Khaki\", shape = \"oval\", fontcolor = \"black\"] \n  \"6\" [label = \"Leaf\nCover: 1\nValue: 0.734999955\", fillcolor = \"Khaki\", shape = \"oval\", fontcolor = \"black\"] \n  \"7\" [label = \"Leaf\nCover: 1\nValue: 0.945000052\", fillcolor = \"Khaki\", shape = \"oval\", fontcolor = \"black\"] \n  \"8\" [label = \"Tree 0\ndose\nCover: 4\nGain: 40.3333321\", fillcolor = \"Beige\", shape = \"rectangle\", fontcolor = \"black\"] \n  \"9\" [label = \"Leaf\nCover: 1\nValue: -1.95000005\", fillcolor = \"Khaki\", shape = \"oval\", fontcolor = \"black\"] \n  \"10\" [label = \"dose\nCover: 3\nGain: 60.1666679\", fillcolor = \"Beige\", shape = \"rectangle\", fontcolor = \"black\"] \n  \"11\" [label = \"dose\nCover: 2\nGain: 0.5\", fillcolor = \"Beige\", shape = \"rectangle\", fontcolor = \"black\"] \n  \"12\" [label = \"Leaf\nCover: 1\nValue: -1.6500001\", fillcolor = \"Khaki\", shape = \"oval\", fontcolor = \"black\"] \n  \"13\" [label = \"Leaf\nCover: 1\nValue: 1.05000007\", fillcolor = \"Khaki\", shape = \"oval\", fontcolor = \"black\"] \n  \"14\" [label = \"Leaf\nCover: 1\nValue: 1.35000002\", fillcolor = \"Khaki\", shape = \"oval\", fontcolor = \"black\"] \n\"1\"->\"2\" [label = \"< 5\", style = \"bold\"] \n\"3\"->\"4\" [label = \"< 15\", style = \"bold\"] \n\"4\"->\"6\" [label = \"< 10\", style = \"bold\"] \n\"8\"->\"9\" [label = \"< 5\", style = \"bold\"] \n\"10\"->\"11\" [label = \"< 15\", style = \"bold\"] \n\"11\"->\"13\" [label = \"< 10\", style = \"bold\"] \n\"1\"->\"3\" [style = \"bold\", style = \"solid\"] \n\"3\"->\"5\" [style = \"solid\", style = \"solid\"] \n\"4\"->\"7\" [style = \"solid\", style = \"solid\"] \n\"8\"->\"10\" [style = \"solid\", style = \"solid\"] \n\"10\"->\"12\" [style = \"solid\", style = \"solid\"] \n\"11\"->\"14\" [style = \"solid\", style = \"solid\"] \n}","config":{"engine":"dot","options":null}},"evals":[],"jsHooks":[]}</script>
</div>
</div>


</section>
</section>

 ]]></description>
  <category>tutorials</category>
  <category>sql</category>
  <category>xgboost</category>
  <category>machine learning</category>
  <guid>https://athospd.github.io/athosdamiani/posts/2020-08-29-xgboost-sql/</guid>
  <pubDate>Sat, 29 Aug 2020 00:00:00 GMT</pubDate>
  <media:content url="https://athospd.github.io/athosdamiani/posts/2020-08-29-xgboost-sql/xgboost-sql.webp" medium="image" type="image/webp"/>
</item>
<item>
  <title>Bechdel: Movies and Sexism</title>
  <dc:creator>Athos Damiani</dc:creator>
  <link>https://athospd.github.io/athosdamiani/posts/2020-06-09-bechdel/</link>
  <description><![CDATA[ 





<p>A friend introduced me to the <a href="https://pt.wikipedia.org/wiki/Teste_de_Bechdel">Bechdel Test</a> (in Portuguese), which checks whether a movie meets three basic rules:</p>
<ol type="1">
<li>At least two (named) women</li>
<li>… who talk to each other</li>
<li>… about something other than a man</li>
</ol>
<p>The test was named after the cartoonist <a href="https://pt.wikipedia.org/wiki/Alison_Bechdel">Alison Bechdel</a> (in Portuguese), who presented this idea in one of her comic strips, and today it is used to measure the degree of gender discrimination in film.</p>
<p>The website <a href="https://bechdeltest.com/">bechdeltest.com</a> lets anyone contribute movies rated on the Bechdel test and provides easy access to a database of 8,440 movies from 1888 to the present day.</p>
<p>A personal two cents (Athos): Imagine the same rules, but swapping man for woman. How many movies do you think would fail the test? My opinion? Less than 1%!</p>
<section id="findings" class="level2">
<h2 class="anchored" data-anchor-id="findings">Findings</h2>
<ul>
<li>Countries like Canada and Japan seem to be improving;</li>
<li>Brazil has little data, but doesn’t stand out positively;</li>
<li>Awards and IMDB ratings are no different between movies that passed the test and movies that didn’t. One could argue that gender discrimination, while always unjustifiable, can’t even rely on the rhetoric that it pays off financially;</li>
</ul>
</section>
<section id="data" class="level2">
<h2 class="anchored" data-anchor-id="data">Data</h2>
<p>The website provides an API to access the data. So bringing it into R is as simple as the line below.</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb1" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb1-1">bechdel <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">fromJSON</span>(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"http://bechdeltest.com/api/v1/getAllMovies"</span>)</span></code></pre></div></div>
</div>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb2" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb2-1"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">glimpse</span>(bechdel)</span></code></pre></div></div>
<div class="cell-output cell-output-stdout">
<pre><code>Rows: 8,440
Columns: 5
$ year   &lt;int&gt; 1888, 1892, 1895, 1895, 1896, 1896, 1896, 1896, 1898, 1898, 189…
$ title  &lt;chr&gt; "Roundhay Garden Scene", "Pauvre Pierrot", "Tables Turned on th…
$ imdbid &lt;chr&gt; "0392728", "0000003", "0000014", "0132134", "0000091", "0000131…
$ rating &lt;int&gt; 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, …
$ id     &lt;int&gt; 8040, 5433, 5444, 6200, 4982, 5406, 5445, 6199, 5411, 5410, 497…</code></pre>
</div>
</div>
<p>PS: notice the <code>imdbid</code> column. It lets us collect information from IMDB for further analysis!</p>
</section>
<section id="rating" class="level2">
<h2 class="anchored" data-anchor-id="rating">Rating</h2>
<p>The rating goes from 0 to 3 and represents how many rules the movie meets. Let’s label it for easier reading.</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb4" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb4-1">bechdel <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> bechdel <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb4-2">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">mutate</span>(</span>
<span id="cb4-3">    <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">indice_bechdel =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">case_when</span>(</span>
<span id="cb4-4">      rating <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">==</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">0</span> <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">~</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Menos de duas mulheres"</span>,</span>
<span id="cb4-5">      rating <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">==</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span> <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">~</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Pelo menos duas mulheres"</span>,</span>
<span id="cb4-6">      rating <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">==</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">2</span> <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">~</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"... que conversam entre si"</span>,</span>
<span id="cb4-7">      rating <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">==</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">3</span> <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">~</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"... sobre algo que não seja homem"</span></span>
<span id="cb4-8">    ),</span>
<span id="cb4-9">    <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">indice_bechdel =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">fct_reorder</span>(indice_bechdel, rating),</span>
<span id="cb4-10">    <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">decada =</span> (year <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%/%</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">10</span>)<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">*</span><span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">10</span></span>
<span id="cb4-11">  )</span></code></pre></div></div>
</div>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb5" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb5-1">bechdel <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span> </span>
<span id="cb5-2">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">count</span>(indice_bechdel, rating) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb5-3">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">mutate</span>(</span>
<span id="cb5-4">    <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">rating_p =</span> scales<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">::</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">percent</span>(n<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">/</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">sum</span>(n), <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">accuracy =</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span>)</span>
<span id="cb5-5">  ) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb5-6">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">ggplot</span>(<span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">aes</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">y =</span> indice_bechdel, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">x =</span> n)) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span></span>
<span id="cb5-7">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">geom_segment</span>(<span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">aes</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">xend =</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">0</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">yend =</span> indice_bechdel)) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span></span>
<span id="cb5-8">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">geom_point</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">size =</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">12</span>) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span></span>
<span id="cb5-9">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">geom_text</span>(<span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">aes</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">label =</span> rating_p), <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">color =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"white"</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">fontface =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"bold"</span>) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span></span>
<span id="cb5-10">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">labs</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">x =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"contagem"</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">y =</span> <span class="cn" style="color: #8f5902;
background-color: null;
font-style: inherit;">NULL</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">title =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Filmes por Rating Bechdel"</span>) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span></span>
<span id="cb5-11">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">theme</span>(</span>
<span id="cb5-12">    <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">panel.grid.minor.y =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">element_blank</span>(),</span>
<span id="cb5-13">    <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">panel.grid.major.y =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">element_blank</span>(),</span>
<span id="cb5-14">    <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">panel.grid.major.x =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">element_blank</span>()</span>
<span id="cb5-15">  )</span></code></pre></div></div>
<div class="cell-output-display">
<div>
<figure class="figure">
<p><img src="https://athospd.github.io/athosdamiani/posts/2020-06-09-bechdel/index_files/figure-html/unnamed-chunk-5-1.png" class="img-fluid figure-img" width="672"></p>
</figure>
</div>
</div>
</div>
</section>
<section id="evolution-over-the-decades" class="level2">
<h2 class="anchored" data-anchor-id="evolution-over-the-decades">Evolution over the Decades</h2>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb6" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb6-1">bechdel_por_decada <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> bechdel <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb6-2">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">count</span>(decada, indice_bechdel, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">.drop =</span> <span class="cn" style="color: #8f5902;
background-color: null;
font-style: inherit;">FALSE</span>)</span>
<span id="cb6-3"></span>
<span id="cb6-4">bechdel_por_decada <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb6-5">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">ggplot</span>(<span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">aes</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">x =</span> decada, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">fill =</span> indice_bechdel, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">y =</span> n)) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span></span>
<span id="cb6-6">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">geom_col</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">position =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"fill"</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">colour =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"black"</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">width =</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">10</span>) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span></span>
<span id="cb6-7">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">geom_text</span>(<span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">aes</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">label =</span> indice_bechdel), <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">position =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"fill"</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">data =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">filter</span>(bechdel_por_decada, decada <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">==</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1950</span>), <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">vjust =</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">2.5</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">colour =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"white"</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">fontface =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"bold"</span>) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span></span>
<span id="cb6-8">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">scale_fill_brewer</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">palette =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"RdBu"</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">guide =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"none"</span>) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span></span>
<span id="cb6-9">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">labs</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">x =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Década"</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">y =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Proporção"</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">title =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Evolução dos Ratings Bechdel pelas Décadas"</span>) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span></span>
<span id="cb6-10">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">scale_y_continuous</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">labels =</span> scales<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">::</span>percent) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span></span>
<span id="cb6-11">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">theme</span>(</span>
<span id="cb6-12">    <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">panel.grid =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">element_blank</span>()</span>
<span id="cb6-13">  ) </span></code></pre></div></div>
<div class="cell-output-display">
<div>
<figure class="figure">
<p><img src="https://athospd.github.io/athosdamiani/posts/2020-06-09-bechdel/index_files/figure-html/unnamed-chunk-6-1.png" class="img-fluid figure-img" width="672"></p>
</figure>
</div>
</div>
</div>
<p>Bechdel Test results have been improving over time, so the outlook is positive. In an increasingly connected world, the hope is that information reaches more and more people and the discrimination ingrained in our culture dissolves.</p>
</section>
<section id="imdb-information" class="level2">
<h2 class="anchored" data-anchor-id="imdb-information">IMDB information</h2>
<p>The <a href="https://www.imdb.com/">IMDB</a> API is very bureaucratic to use. I found the <a href="http://www.omdbapi.com/">OMDB</a> website, which was much easier to use, though it may have less cool information. Here’s how to use it:</p>
<section id="api-key" class="level3">
<h3 class="anchored" data-anchor-id="api-key">API Key</h3>
<p>First I got an API KEY on their website and put it in my .Renviron, because then I can use <code>Sys.get()</code> to use it without having to show it to anyone.</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb7" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb7-1"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">Sys.getenv</span>(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">'OMDB_APIKEY'</span>)</span>
<span id="cb7-2"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># [1] as48732f1d (just pretend)</span></span></code></pre></div></div>
</div>
</section>
<section id="calling-the-api" class="level3">
<h3 class="anchored" data-anchor-id="calling-the-api">Calling the API</h3>
<p>Then I called the OMDB API using both my API KEY and the IMDB ID. The call has to look like this: <code>http://www.omdbapi.com/?apikey={Sys.getenv('OMDB_APIKEY')}&amp;i=tt{imdbid}</code>. It returns a json that can be easily formatted with <code>jsonlite::fromJSON()</code>.</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb8" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb8-1"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">library</span>(glue)</span>
<span id="cb8-2">pega_info_imdb <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">function</span>(imdbid) {</span>
<span id="cb8-3">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">glue</span>(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"http://www.omdbapi.com/?apikey={Sys.getenv('OMDB_APIKEY')}&amp;i=tt{imdbid}"</span>) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb8-4">    <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">fromJSON</span>() <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb8-5">    <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">as_tibble</span>()</span>
<span id="cb8-6">}</span>
<span id="cb8-7"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># safely because the API may return error 522 (on the OMDB server)</span></span>
<span id="cb8-8">pega_info_imdb_safe <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">safely</span>(pega_info_imdb)</span>
<span id="cb8-9"></span>
<span id="cb8-10"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># takes ~40 min</span></span>
<span id="cb8-11">bechdel_com_imdb <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> bechdel <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb8-12">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">as_tibble</span>() <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb8-13">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">mutate</span>(</span>
<span id="cb8-14">    <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">infos_imdb =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">map</span>(imdbid, pega_info_imdb_safe)</span>
<span id="cb8-15">  )</span></code></pre></div></div>
</div>
</section>
<section id="tidying-the-data" class="level3">
<h3 class="anchored" data-anchor-id="tidying-the-data">Tidying the data</h3>
<p>The data needs some massaging first because it comes as a nested list, all as text.</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb9" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb9-1"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Tidies the nested tables</span></span>
<span id="cb9-2">arruma_infos_imdb <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">function</span>(infos_imdb) {</span>
<span id="cb9-3">  <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">if</span>(<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">!</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">is.null</span>(infos_imdb<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span>Ratings)) {</span>
<span id="cb9-4">    infos_imdb <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span> </span>
<span id="cb9-5">      <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">nest</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">Ratings =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">c</span>(Ratings)) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb9-6">      <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">mutate</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">Ratings =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">map</span>(Ratings, <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">~</span>.x<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span>Ratings <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">filter</span>(Source <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%in%</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Internet Movie Database"</span>))) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb9-7">      <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">unnest</span>(Ratings)</span>
<span id="cb9-8">  } <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">else</span> {</span>
<span id="cb9-9">    infos_imdb</span>
<span id="cb9-10">  }</span>
<span id="cb9-11">}</span>
<span id="cb9-12"></span>
<span id="cb9-13"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># creates bechdel with the imdb info and fixes the variable types</span></span>
<span id="cb9-14">bechdel_com_imdb <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> bechdel_com_imdb <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb9-15">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">mutate</span>(</span>
<span id="cb9-16">    <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">deu_errado =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">map_lgl</span>(infos_imdb, <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">~</span> <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">!</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">is.null</span>(.x<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span>error)),</span>
<span id="cb9-17">    <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">infos_imdb =</span> infos_imdb <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">map</span>(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"result"</span>) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">map</span>(arruma_infos_imdb)</span>
<span id="cb9-18">  ) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb9-19">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">unnest</span>(infos_imdb) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb9-20">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">mutate</span>(</span>
<span id="cb9-21">    <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">imdbRating_num =</span> imdbRating <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">parse_number</span>()</span>
<span id="cb9-22">  )</span></code></pre></div></div>
</div>
</section>
<section id="final-table" class="level3">
<h3 class="anchored" data-anchor-id="final-table">Final table</h3>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb10" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb10-1"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">glimpse</span>(bechdel_com_imdb)</span></code></pre></div></div>
<div class="cell-output cell-output-stdout">
<pre><code>Rows: 8,414
Columns: 41
$ year           &lt;int&gt; 1888, 1892, 1895, 1895, 1896, 1896, 1896, 1896, 1898, 1…
$ title          &lt;chr&gt; "Roundhay Garden Scene", "Pauvre Pierrot", "Tables Turn…
$ imdbid         &lt;chr&gt; "0392728", "0000003", "0000014", "0132134", "0000091", …
$ rating         &lt;int&gt; 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 3, 0, 0, 0, 0, 0, 0, 0…
$ id             &lt;int&gt; 8040, 5433, 5444, 6200, 4982, 5406, 5445, 6199, 5411, 5…
$ indice_bechdel &lt;fct&gt; Menos de duas mulheres, Menos de duas mulheres, Menos d…
$ decada         &lt;dbl&gt; 1880, 1890, 1890, 1890, 1890, 1890, 1890, 1890, 1890, 1…
$ Title          &lt;chr&gt; "Roundhay Garden Scene", "Poor Pierrot", "The Sprinkler…
$ Year           &lt;chr&gt; "1888", "1892", "1895", "1895", "1896", "1896", "1896",…
$ Rated          &lt;chr&gt; "NOT RATED", "N/A", "Not Rated", "Not Rated", "N/A", "N…
$ Released       &lt;chr&gt; "14 Oct 1888", "28 Oct 1892", "N/A", "28 Aug 1895", "N/…
$ Runtime        &lt;chr&gt; "1 min", "4 min", "1 min", "1 min", "3 min", "1 min", "…
$ Genre          &lt;chr&gt; "Documentary, Short", "Animation, Comedy, Short, Romanc…
$ Director       &lt;chr&gt; "Louis Aimé Augustin Le Prince", "Émile Reynaud", "Loui…
$ Writer         &lt;chr&gt; "N/A", "N/A", "N/A", "N/A", "Georges Méliès", "N/A", "A…
$ Actors         &lt;chr&gt; "Annie Hartley, Adolphe Le Prince, Joseph Whitley, Sara…
$ Plot           &lt;chr&gt; "In the garden, a man asks his friends to do something …
$ Language       &lt;chr&gt; "N/A", "N/A", "N/A", "N/A", "N/A", "None", "N/A", "N/A"…
$ Country        &lt;chr&gt; "UK, France", "France", "France", "USA", "France", "Fra…
$ Awards         &lt;chr&gt; "N/A", "N/A", "N/A", "N/A", "N/A", "N/A", "N/A", "N/A",…
$ Poster         &lt;chr&gt; "https://m.media-amazon.com/images/M/MV5BOGE3YjczMTQtZT…
$ Metascore      &lt;chr&gt; "N/A", "N/A", "N/A", "N/A", "N/A", "N/A", "N/A", "N/A",…
$ imdbRating     &lt;chr&gt; "7.5", "6.6", "7.1", "6.7", "6.7", "5.8", "5.2", "7.4",…
$ imdbVotes      &lt;chr&gt; "4,609", "1,144", "4,287", "1,755", "2,149", "1,040", "…
$ imdbID         &lt;chr&gt; "tt0392728", "tt0000003", "tt0000014", "tt0132134", "tt…
$ Type           &lt;chr&gt; "movie", "movie", "movie", "movie", "movie", "movie", "…
$ DVD            &lt;chr&gt; "N/A", "N/A", "N/A", "N/A", "N/A", "N/A", "N/A", "N/A",…
$ BoxOffice      &lt;chr&gt; "N/A", "N/A", "N/A", "N/A", "N/A", "N/A", "N/A", "N/A",…
$ Production     &lt;chr&gt; "N/A", "N/A", "N/A", "N/A", "N/A", "N/A", "N/A", "N/A",…
$ Website        &lt;chr&gt; "N/A", "N/A", "N/A", "N/A", "N/A", "N/A", "N/A", "N/A",…
$ Response       &lt;chr&gt; "True", "True", "True", "True", "True", "True", "True",…
$ Source         &lt;chr&gt; "Internet Movie Database", "Internet Movie Database", "…
$ Value          &lt;chr&gt; "7.5/10", "6.6/10", "7.1/10", "6.7/10", "6.7/10", "5.8/…
$ Error          &lt;chr&gt; NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA,…
$ Season         &lt;chr&gt; NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA,…
$ Episode        &lt;chr&gt; NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA,…
$ seriesID       &lt;chr&gt; NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA,…
$ totalSeasons   &lt;chr&gt; NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA,…
$ Ratings        &lt;lgl&gt; NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA,…
$ deu_errado     &lt;lgl&gt; FALSE, FALSE, FALSE, FALSE, FALSE, FALSE, FALSE, FALSE,…
$ imdbRating_num &lt;dbl&gt; 7.5, 6.6, 7.1, 6.7, 6.7, 5.8, 5.2, 7.4, 6.0, 7.5, 7.5, …</code></pre>
</div>
</div>
</section>
<section id="bechdel-by-country" class="level3">
<h3 class="anchored" data-anchor-id="bechdel-by-country">Bechdel by Country</h3>
<p>Japan and Canada are the countries with the clearest trends of change. The others seem stagnant. Brazil has 45 movies analyzed and doesn’t look different from Germany, the UK, the US or France.</p>
<div class="cell">
<div class="cell-output-display">
<div>
<figure class="figure">
<p><img src="https://athospd.github.io/athosdamiani/posts/2020-06-09-bechdel/index_files/figure-html/unnamed-chunk-13-1.png" class="img-fluid figure-img" width="672"></p>
</figure>
</div>
</div>
</div>
<section id="brazilian-movies" class="level4">
<h4 class="anchored" data-anchor-id="brazilian-movies">Brazilian movies</h4>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb12" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb12-1">bechdel_com_imdb <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb12-2">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">filter</span>(<span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">str_detect</span>(Country, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Brazil"</span>)) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb12-3">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">select</span>(year, Title, rating) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb12-4">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">arrange</span>(rating) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb12-5">  knitr<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">::</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">kable</span>()</span></code></pre></div></div>
<div class="cell-output-display">
<table class="caption-top table table-sm table-striped small">
<thead>
<tr class="header">
<th style="text-align: right;">year</th>
<th style="text-align: left;">Title</th>
<th style="text-align: right;">rating</th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td style="text-align: right;">2007</td>
<td style="text-align: left;">Estomago: A Gastronomic Story</td>
<td style="text-align: right;">0</td>
</tr>
<tr class="even">
<td style="text-align: right;">2007</td>
<td style="text-align: left;">You, Me and Him</td>
<td style="text-align: right;">0</td>
</tr>
<tr class="odd">
<td style="text-align: right;">2009</td>
<td style="text-align: left;">The Secret of Kells</td>
<td style="text-align: right;">0</td>
</tr>
<tr class="even">
<td style="text-align: right;">2011</td>
<td style="text-align: left;">Samsara</td>
<td style="text-align: right;">0</td>
</tr>
<tr class="odd">
<td style="text-align: right;">2013</td>
<td style="text-align: left;">Boy and the World</td>
<td style="text-align: right;">0</td>
</tr>
<tr class="even">
<td style="text-align: right;">2014</td>
<td style="text-align: left;">Ardor</td>
<td style="text-align: right;">0</td>
</tr>
<tr class="odd">
<td style="text-align: right;">2002</td>
<td style="text-align: left;">City of God</td>
<td style="text-align: right;">1</td>
</tr>
<tr class="even">
<td style="text-align: right;">2003</td>
<td style="text-align: left;">Fear X</td>
<td style="text-align: right;">1</td>
</tr>
<tr class="odd">
<td style="text-align: right;">2011</td>
<td style="text-align: left;">Rio</td>
<td style="text-align: right;">1</td>
</tr>
<tr class="even">
<td style="text-align: right;">2012</td>
<td style="text-align: left;">Two Rabbits</td>
<td style="text-align: right;">1</td>
</tr>
<tr class="odd">
<td style="text-align: right;">2013</td>
<td style="text-align: left;">Vai que dá Certo</td>
<td style="text-align: right;">1</td>
</tr>
<tr class="even">
<td style="text-align: right;">2014</td>
<td style="text-align: left;">Trash</td>
<td style="text-align: right;">1</td>
</tr>
<tr class="odd">
<td style="text-align: right;">2014</td>
<td style="text-align: left;">Futuro Beach</td>
<td style="text-align: right;">1</td>
</tr>
<tr class="even">
<td style="text-align: right;">2018</td>
<td style="text-align: left;">Tito and the Birds</td>
<td style="text-align: right;">1</td>
</tr>
<tr class="odd">
<td style="text-align: right;">2019</td>
<td style="text-align: left;">Bacurau</td>
<td style="text-align: right;">1</td>
</tr>
<tr class="even">
<td style="text-align: right;">1985</td>
<td style="text-align: left;">Kiss of the Spider Woman</td>
<td style="text-align: right;">2</td>
</tr>
<tr class="odd">
<td style="text-align: right;">2003</td>
<td style="text-align: left;">The Man Who Copied</td>
<td style="text-align: right;">2</td>
</tr>
<tr class="even">
<td style="text-align: right;">2009</td>
<td style="text-align: left;">From Beginning to End</td>
<td style="text-align: right;">2</td>
</tr>
<tr class="odd">
<td style="text-align: right;">2019</td>
<td style="text-align: left;">Turma da Mônica: Laços</td>
<td style="text-align: right;">2</td>
</tr>
<tr class="even">
<td style="text-align: right;">1986</td>
<td style="text-align: left;">Hour of the Star</td>
<td style="text-align: right;">3</td>
</tr>
<tr class="odd">
<td style="text-align: right;">1998</td>
<td style="text-align: left;">Central Station</td>
<td style="text-align: right;">3</td>
</tr>
<tr class="even">
<td style="text-align: right;">2004</td>
<td style="text-align: left;">Nina</td>
<td style="text-align: right;">3</td>
</tr>
<tr class="odd">
<td style="text-align: right;">2005</td>
<td style="text-align: left;">Æon Flux</td>
<td style="text-align: right;">3</td>
</tr>
<tr class="even">
<td style="text-align: right;">2007</td>
<td style="text-align: left;">Elite Squad</td>
<td style="text-align: right;">3</td>
</tr>
<tr class="odd">
<td style="text-align: right;">2008</td>
<td style="text-align: left;">Blindness</td>
<td style="text-align: right;">3</td>
</tr>
<tr class="even">
<td style="text-align: right;">2008</td>
<td style="text-align: left;">Lion’s Den</td>
<td style="text-align: right;">3</td>
</tr>
<tr class="odd">
<td style="text-align: right;">2010</td>
<td style="text-align: left;">So Hard to Forget</td>
<td style="text-align: right;">3</td>
</tr>
<tr class="even">
<td style="text-align: right;">2011</td>
<td style="text-align: left;">Fast Five</td>
<td style="text-align: right;">3</td>
</tr>
<tr class="odd">
<td style="text-align: right;">2011</td>
<td style="text-align: left;">Found Memories</td>
<td style="text-align: right;">3</td>
</tr>
<tr class="even">
<td style="text-align: right;">2012</td>
<td style="text-align: left;">Neighboring Sounds</td>
<td style="text-align: right;">3</td>
</tr>
<tr class="odd">
<td style="text-align: right;">2012</td>
<td style="text-align: left;">Tabu</td>
<td style="text-align: right;">3</td>
</tr>
<tr class="even">
<td style="text-align: right;">2012</td>
<td style="text-align: left;">Snezhnaya koroleva</td>
<td style="text-align: right;">3</td>
</tr>
<tr class="odd">
<td style="text-align: right;">2013</td>
<td style="text-align: left;">Reaching for the Moon</td>
<td style="text-align: right;">3</td>
</tr>
<tr class="even">
<td style="text-align: right;">2013</td>
<td style="text-align: left;">Open Road</td>
<td style="text-align: right;">3</td>
</tr>
<tr class="odd">
<td style="text-align: right;">2013</td>
<td style="text-align: left;">My Mom Is a Character</td>
<td style="text-align: right;">3</td>
</tr>
<tr class="even">
<td style="text-align: right;">2013</td>
<td style="text-align: left;">Brazilian Western</td>
<td style="text-align: right;">3</td>
</tr>
<tr class="odd">
<td style="text-align: right;">2013</td>
<td style="text-align: left;">Time and the Wind</td>
<td style="text-align: right;">3</td>
</tr>
<tr class="even">
<td style="text-align: right;">2014</td>
<td style="text-align: left;">August Winds</td>
<td style="text-align: right;">3</td>
</tr>
<tr class="odd">
<td style="text-align: right;">2014</td>
<td style="text-align: left;">The Way He Looks</td>
<td style="text-align: right;">3</td>
</tr>
<tr class="even">
<td style="text-align: right;">2014</td>
<td style="text-align: left;">Last Night</td>
<td style="text-align: right;">3</td>
</tr>
<tr class="odd">
<td style="text-align: right;">2015</td>
<td style="text-align: left;">Neon Bull</td>
<td style="text-align: right;">3</td>
</tr>
<tr class="even">
<td style="text-align: right;">2015</td>
<td style="text-align: left;">The Second Mother</td>
<td style="text-align: right;">3</td>
</tr>
<tr class="odd">
<td style="text-align: right;">2016</td>
<td style="text-align: left;">Aquarius</td>
<td style="text-align: right;">3</td>
</tr>
<tr class="even">
<td style="text-align: right;">2017</td>
<td style="text-align: left;">Call Me by Your Name</td>
<td style="text-align: right;">3</td>
</tr>
<tr class="odd">
<td style="text-align: right;">2018</td>
<td style="text-align: left;">The Heiresses</td>
<td style="text-align: right;">3</td>
</tr>
</tbody>
</table>
</div>
</div>
</section>
</section>
<section id="award-winning-movies" class="level3">
<h3 class="anchored" data-anchor-id="award-winning-movies">Award-winning Movies</h3>
<p>The dataset includes awards information. Movies without awards don’t stand apart from award-winning movies in terms of Bechdel rating, which indicates that sexism is not related to a movie’s success.</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb13" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb13-1">bechdel_por_decada_e_premiacao <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> bechdel_com_imdb <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb13-2">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">mutate</span>(</span>
<span id="cb13-3">    <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">premiacao =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">case_when</span>(</span>
<span id="cb13-4">      <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">str_detect</span>(Awards, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"N/A"</span>) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">|</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">is.na</span>(Awards) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">~</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"z Sem Premiação"</span>,</span>
<span id="cb13-5">      <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">str_detect</span>(Awards, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Won.*Oscar"</span>) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">~</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"a Vencedor de Oscar"</span>,</span>
<span id="cb13-6">      <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">str_detect</span>(Awards, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Won.*(BAFTA|Golden Globe)"</span>) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">~</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"b Vencedor de BAFTA/Golden Globe"</span>,</span>
<span id="cb13-7">      <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">str_detect</span>(Awards, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Nominated"</span>) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">~</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"c Nomeado ao Oscar/BAFTA/Golden Globe"</span>,</span>
<span id="cb13-8">      <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">str_detect</span>(Awards, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"win"</span>) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">~</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"d Vencedor de outros prêmios"</span>,</span>
<span id="cb13-9">      <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">str_detect</span>(Awards, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"nomination"</span>) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">~</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"d Nomeado a outros prêmios"</span>,</span>
<span id="cb13-10">      <span class="cn" style="color: #8f5902;
background-color: null;
font-style: inherit;">TRUE</span> <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">~</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"e Outro"</span></span>
<span id="cb13-11">    )</span>
<span id="cb13-12">  ) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb13-13">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">count</span>(premiacao, decada, indice_bechdel, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">.drop =</span> <span class="cn" style="color: #8f5902;
background-color: null;
font-style: inherit;">FALSE</span>)</span>
<span id="cb13-14">  </span>
<span id="cb13-15">bechdel_por_decada_e_premiacao <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb13-16">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">filter</span>(decada <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&gt;=</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1940</span>) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb13-17">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">ggplot</span>(<span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">aes</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">x =</span> decada, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">fill =</span> indice_bechdel, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">y =</span> n)) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span></span>
<span id="cb13-18">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">geom_col</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">position =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"fill"</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">colour =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"black"</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">width =</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">10</span>) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span></span>
<span id="cb13-19">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">scale_fill_brewer</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">palette =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"RdBu"</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">guide =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"none"</span>) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span></span>
<span id="cb13-20">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">labs</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">x =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Década"</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">y =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Proporção"</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">title =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Evolução dos Ratings Bechdel pelas Décadas"</span>) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span></span>
<span id="cb13-21">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">scale_y_continuous</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">labels =</span> scales<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">::</span>percent) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span></span>
<span id="cb13-22">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">theme</span>(</span>
<span id="cb13-23">    <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">panel.grid =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">element_blank</span>()</span>
<span id="cb13-24">  ) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span></span>
<span id="cb13-25">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">facet_wrap</span>(<span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">vars</span>(premiacao), <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">ncol =</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">3</span>)</span></code></pre></div></div>
<div class="cell-output-display">
<div>
<figure class="figure">
<p><img src="https://athospd.github.io/athosdamiani/posts/2020-06-09-bechdel/index_files/figure-html/unnamed-chunk-15-1.png" class="img-fluid figure-img" width="672"></p>
</figure>
</div>
</div>
</div>
</section>
<section id="imdb-rating" class="level3">
<h3 class="anchored" data-anchor-id="imdb-rating">IMDB rating</h3>
<p>IMDB ratings also don’t differ between movies that passed and movies that failed the Bechdel test.</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb14" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb14-1">bechdel_por_decada_e_pais <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> bechdel_por_decada_e_pais <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb14-2">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">mutate</span>(</span>
<span id="cb14-3">    <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">imdbVotes =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">parse_number</span>(imdbVotes),</span>
<span id="cb14-4">    <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">passou_no_teste =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">case_when</span>(</span>
<span id="cb14-5">      rating <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%in%</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">3</span> <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">~</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Sim"</span>,</span>
<span id="cb14-6">      <span class="cn" style="color: #8f5902;
background-color: null;
font-style: inherit;">TRUE</span> <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">~</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Não"</span></span>
<span id="cb14-7">    )</span>
<span id="cb14-8">  )</span>
<span id="cb14-9">bechdel_nota_por_decada <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> bechdel_por_decada_e_pais <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb14-10">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">group_by</span>(pais) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb14-11">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">filter</span>(<span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">n</span>() <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&gt;</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">10</span>) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb14-12">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">group_by</span>(pais, decada, passou_no_teste) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb14-13">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">summarise</span>(</span>
<span id="cb14-14">    <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">imdb_media =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">mean</span>(imdbRating_num, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">na.rm =</span> <span class="cn" style="color: #8f5902;
background-color: null;
font-style: inherit;">TRUE</span>),</span>
<span id="cb14-15">    <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">imdb_dp =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">sd</span>(imdbRating_num, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">na.rm =</span> <span class="cn" style="color: #8f5902;
background-color: null;
font-style: inherit;">TRUE</span>),</span>
<span id="cb14-16">    <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">imdb_inf =</span> imdb_media <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">-</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">2</span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">*</span>imdb_dp,</span>
<span id="cb14-17">    <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">imdb_sup =</span> imdb_media <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">2</span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">*</span>imdb_dp</span>
<span id="cb14-18">  )</span>
<span id="cb14-19"></span>
<span id="cb14-20">bechdel_nota_por_decada <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span> </span>
<span id="cb14-21">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">filter</span>(decada <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&gt;=</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1940</span>) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb14-22">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">ggplot</span>(<span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">aes</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">x =</span> decada, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">y =</span> imdb_media, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">colour =</span> passou_no_teste)) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span></span>
<span id="cb14-23">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">geom_line</span>() <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span></span>
<span id="cb14-24">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">geom_point</span>() <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span></span>
<span id="cb14-25">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">geom_jitter</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">data =</span> bechdel_por_decada_e_pais <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">filter</span>(decada <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&gt;=</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1940</span>), <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">aes</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">y =</span> imdbRating_num), <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">alpha =</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0.05</span>) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span></span>
<span id="cb14-26">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">geom_ribbon</span>(<span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">aes</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">ymin =</span> imdb_inf, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">ymax =</span> imdb_sup, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">fill =</span> passou_no_teste), <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">alpha =</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0.1</span>) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span></span>
<span id="cb14-27">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">facet_wrap</span>(<span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">vars</span>(pais)) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span></span>
<span id="cb14-28">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">labs</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">x =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Década"</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">y =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Nota IMDB"</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">colour =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Passou no Teste"</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">fill =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Passou no Teste"</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">title =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Notas do IMDB vs Teste de Bechdel"</span>) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span></span>
<span id="cb14-29">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">scale_x_continuous</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">breaks =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">c</span>(<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1950</span>, <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">2020</span>)) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span></span>
<span id="cb14-30">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">theme</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">legend.position =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"top"</span>)</span></code></pre></div></div>
<div class="cell-output-display">
<div>
<figure class="figure">
<p><img src="https://athospd.github.io/athosdamiani/posts/2020-06-09-bechdel/index_files/figure-html/unnamed-chunk-16-1.png" class="img-fluid figure-img" width="672"></p>
</figure>
</div>
</div>
</div>
<p>What did you think of the analyses? Let us know! It’s a topic worth discussing by everyone.</p>


</section>
</section>

 ]]></description>
  <category>analysis</category>
  <category>ggplot2</category>
  <category>tidyverse</category>
  <category>api</category>
  <category>imdb</category>
  <guid>https://athospd.github.io/athosdamiani/posts/2020-06-09-bechdel/</guid>
  <pubDate>Tue, 09 Jun 2020 00:00:00 GMT</pubDate>
  <media:content url="https://athospd.github.io/athosdamiani/posts/2020-06-09-bechdel/bechdel.webp" medium="image" type="image/webp"/>
</item>
<item>
  <title>Data Challenge - Image recovery</title>
  <dc:creator>Athos Damiani</dc:creator>
  <dc:creator>Daniel Falbel</dc:creator>
  <dc:creator>Bruna Wundervald</dc:creator>
  <link>https://athospd.github.io/athosdamiani/posts/2018-08-09-desafio-recuperacao-imagem/</link>
  <description><![CDATA[ 





<p>Files with the <em>.jpg</em> extension store 5 coordinates, which are enough for the computer to understand and draw the image:</p>
<ul>
<li><code>x</code> and <code>y</code>: the cartesian coordinates of the image; and</li>
<li><code>r</code>, <code>g</code> and <code>b</code>: red, green and blue, respectively, which together form colors.</li>
</ul>
<p>Every color we know can be made by combining these three colors. The intensity of each color ranges from 0 to 1. For each point on the (x,y) plane there is an associated color. So an image can be represented by a dataset with 5 columns (<code>x</code>, <code>y</code>, <code>r</code>, <code>g</code> and <code>b</code>) whose number of rows is the number of pixels in the image.</p>
<section id="goals" class="level1">
<h1>Goals</h1>
<ul>
<li>Find out which model, linear regression or decision trees, is the most suitable to recover the blue component of the <a href="purple_wave.webp">purple_wave.jpg image.</a></li>
</ul>
<p><img src="https://athospd.github.io/athosdamiani/posts/2018-08-09-desafio-recuperacao-imagem/purple_wave.webp" class="img-fluid"></p>
<ul>
<li>Find out which model, linear regression or decision trees, is the most suitable to recover the blue component of the <a href="xadrez_colorido.webp">xadrez_colorido.jpg image.</a></li>
</ul>
<p><img src="https://athospd.github.io/athosdamiani/posts/2018-08-09-desafio-recuperacao-imagem/xadrez_colorido.webp" class="img-fluid"></p>
</section>
<section id="how-to-start" class="level1">
<h1>How to start?</h1>
<p>With the help of the <code>readJPEG()</code> function from the <code>jpeg</code> package, load one of the images into R and turn it into a data.frame:</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb1" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb1-1"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">library</span>(tidyverse)</span>
<span id="cb1-2"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">library</span>(jpeg)</span>
<span id="cb1-3"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># loads a jpeg image into R</span></span>
<span id="cb1-4"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">download.file</span>(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"http://curso-r.comimages/posts/conteudo/desafio-recuperacao-img/purple_wave.webp"</span>, </span>
<span id="cb1-5">              <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">destfile =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"purple_wave.jpg"</span>)</span>
<span id="cb1-6"></span>
<span id="cb1-7">img <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">readJPEG</span>(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"purple_wave.jpg"</span>)</span>
<span id="cb1-8"></span>
<span id="cb1-9"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># turns the image array into a data.frame with position (x,y)</span></span>
<span id="cb1-10"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># and color (r,g,b) info; image dimensions</span></span>
<span id="cb1-11">img_dim <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">dim</span>(img)</span>
<span id="cb1-12"></span>
<span id="cb1-13"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># RGB to data.frame</span></span>
<span id="cb1-14">img_df <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">data.frame</span>(</span>
<span id="cb1-15">  <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">x =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">rep</span>(<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">:</span>img_dim[<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">2</span>], <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">each =</span> img_dim[<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span>]),</span>
<span id="cb1-16">  <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">y =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">rep</span>(img_dim[<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span>]<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">:</span><span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span>, img_dim[<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">2</span>]),</span>
<span id="cb1-17">  <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">r =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">as.vector</span>(img[,,<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span>]),</span>
<span id="cb1-18">  <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">g =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">as.vector</span>(img[,,<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">2</span>]),</span>
<span id="cb1-19">  <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">b =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">as.vector</span>(img[,,<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">3</span>])</span>
<span id="cb1-20">) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb1-21">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">mutate</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">cor =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">rgb</span>(r, g, b),</span>
<span id="cb1-22">         <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">id =</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">:</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">n</span>())</span></code></pre></div></div>
</div>
<p>Split the data.frame into two parts: one with the blue (column b) and one without. - Part 1) <code>x</code>, <code>y</code>, <code>r</code>, <code>g</code> - Part 2) <code>x</code>, <code>y</code>, <code>r</code>, <code>g</code>, <code>b</code></p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb2" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb2-1"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># for reproducibility</span></span>
<span id="cb2-2"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">set.seed</span>(<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">2018</span>) </span>
<span id="cb2-3"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Part 1) x, y, r, g</span></span>
<span id="cb2-4">img_df_parte1 <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> img_df <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span> </span>
<span id="cb2-5">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">sample_frac</span>(<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">3</span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">/</span><span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">5</span>) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span> <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># taking 3/5 of the dataset</span></span>
<span id="cb2-6">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">mutate</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">b_backup =</span> b, <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># backup of the original blue</span></span>
<span id="cb2-7">         <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">b =</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">0</span>, <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># removing the blue from the image</span></span>
<span id="cb2-8">         <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">cor =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">rgb</span>(r, g, b)) <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># image color without the blue</span></span>
<span id="cb2-9"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Part 2) x, y, r, g, b</span></span>
<span id="cb2-10">img_df_parte2 <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> img_df <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">filter</span>(<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">!</span>id<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%in%</span>img_df_parte1<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span>id)</span></code></pre></div></div>
</div>
<p>Visualizing:</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb3" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb3-1"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Image without the blue</span></span>
<span id="cb3-2"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">ggplot</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">data =</span> img_df_parte1, <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">aes</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">x =</span> x, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">y =</span> y)) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span> </span>
<span id="cb3-3">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">geom_point</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">colour =</span> img_df_parte1<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span>cor) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span></span>
<span id="cb3-4">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">labs</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">x =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"x"</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">y =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"y"</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">title =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Imagem sem B (azul)"</span>) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span></span>
<span id="cb3-5">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">coord_fixed</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">ratio =</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span>) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span></span>
<span id="cb3-6">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">theme_bw</span>()</span></code></pre></div></div>
<div class="cell-output-display">
<div>
<figure class="figure">
<p><img src="https://athospd.github.io/athosdamiani/posts/2018-08-09-desafio-recuperacao-imagem/index_files/figure-html/unnamed-chunk-4-1.png" class="img-fluid figure-img" width="672"></p>
</figure>
</div>
</div>
</div>
<p>Your task is to recover the blue (which we erased from the image) in Part 1, using predictive models built with Part 2 (which still has the blue!) for both images (<code>xadrez_colorido</code> and <code>purple_wave</code>).</p>
</section>
<section id="result" class="level1">
<h1>Result</h1>
<p>The result of this data challenge will be published on the blog on August 24! Good luck!</p>


</section>

 ]]></description>
  <category>announcements</category>
  <category>app</category>
  <category>packages</category>
  <guid>https://athospd.github.io/athosdamiani/posts/2018-08-09-desafio-recuperacao-imagem/</guid>
  <pubDate>Thu, 09 Aug 2018 00:00:00 GMT</pubDate>
  <media:content url="https://athospd.github.io/athosdamiani/posts/2018-08-09-desafio-recuperacao-imagem/purple_wave.webp" medium="image" type="image/webp"/>
</item>
<item>
  <title>Are tree-based models immune to multicollinearity?</title>
  <dc:creator>Athos Damiani</dc:creator>
  <dc:creator>Julio Trecenti</dc:creator>
  <link>https://athospd.github.io/athosdamiani/posts/2018-05-22-arvore-e-multicolinearidade/</link>
  <description><![CDATA[ 





<p>Tree-based models like decision trees, random forest, ligthGBM and xgboost are known, among other qualities, for their robustness to multicollinearity. It’s well known that their predictive power isn’t shaken by the presence of extremely correlated variables.</p>
<p>But who has never used a Random Forest for variable selection? Taking, for example, the top 10 most important ones and dropping the rest?</p>
<p>Or even ventured an interpretation and drew conclusions from the order of the most important variables?</p>
<p>Below we show why we shouldn’t ignore multicollinearity completely!</p>
<section id="a-nice-little-model" class="level2">
<h2 class="anchored" data-anchor-id="a-nice-little-model">A nice little model</h2>
<p>First let’s fit a nice little model, free of multicollinearity. Suppose we want to predict <code>Petal.Length</code> using the sepal measurements (<code>Sepal.Width</code> and <code>Sepal.Length</code>) from our good old <code>iris</code> dataset.</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb1" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb1-1"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">library</span>(tidyverse)</span>
<span id="cb1-2"></span>
<span id="cb1-3">iris2 <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> iris <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">select</span>(Sepal.Length, Sepal.Width, Petal.Length)</span>
<span id="cb1-4">iris2 <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span> cor <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span> corrplot<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">::</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">corrplot</span>()</span></code></pre></div></div>
<div class="cell-output-display">
<div>
<figure class="figure">
<p><img src="https://athospd.github.io/athosdamiani/posts/2018-05-22-arvore-e-multicolinearidade/index_files/figure-html/unnamed-chunk-2-1.png" class="img-fluid figure-img" width="672"></p>
</figure>
</div>
</div>
</div>
<p>The plot above shows that the explanatory variables are not strongly correlated. Fitting a random fores, we get the following variable importance order:</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb2" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb2-1"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">library</span>(randomForest)</span>
<span id="cb2-2">iris2_rf <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">randomForest</span>(Petal.Length <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">~</span> ., <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">data =</span> iris2)</span>
<span id="cb2-3"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">varImpPlot</span>(iris2_rf)</span></code></pre></div></div>
<div class="cell-output-display">
<div>
<figure class="figure">
<p><img src="https://athospd.github.io/athosdamiani/posts/2018-05-22-arvore-e-multicolinearidade/index_files/figure-html/unnamed-chunk-3-1.png" class="img-fluid figure-img" width="672"></p>
</figure>
</div>
</div>
</div>
<p>No surprises. Now on to the problem!</p>
</section>
<section id="an-ugly-model" class="level2">
<h2 class="anchored" data-anchor-id="an-ugly-model">An ugly model</h2>
<p>Let’s forge an extreme situation where many variables are multicollinear. I’ll do this by repeating the <code>Sepal.Length</code> column several times.</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb3" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb3-1">iris3 <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">accumulate</span>(<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">:</span><span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">20</span>, <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">~</span>{</span>
<span id="cb3-2">  .x[[<span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">paste0</span>(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Sepal.Length"</span>, .y)]] <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span>  iris2<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span>Sepal.Length</span>
<span id="cb3-3">  .x</span>
<span id="cb3-4">}, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">.init =</span> iris2)</span>
<span id="cb3-5"></span>
<span id="cb3-6">iris3[[<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">20</span>]] <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span> cor <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span> corrplot<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">::</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">corrplot</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">order =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"alphabet"</span>)</span></code></pre></div></div>
<div class="cell-output-display">
<div>
<figure class="figure">
<p><img src="https://athospd.github.io/athosdamiani/posts/2018-05-22-arvore-e-multicolinearidade/index_files/figure-html/unnamed-chunk-4-1.png" class="img-fluid figure-img" width="672"></p>
</figure>
</div>
</div>
</div>
<p>Now things are ugly! We have 20 perfectly collinear variables. Even so, a random forest on this new dataset wouldn’t lose predictive power.</p>
<p>But what happened to the variable importance?</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb4" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb4-1">iris3_rf <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">randomForest</span>(Petal.Length <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">~</span> ., <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">data =</span> iris3[[<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">20</span>]])</span>
<span id="cb4-2"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">varImpPlot</span>(iris3_rf)</span></code></pre></div></div>
<div class="cell-output-display">
<div>
<figure class="figure">
<p><img src="https://athospd.github.io/athosdamiani/posts/2018-05-22-arvore-e-multicolinearidade/index_files/figure-html/unnamed-chunk-5-1.png" class="img-fluid figure-img" width="672"></p>
</figure>
</div>
</div>
</div>
<p>Here the tables have turned: we would conclude that <code>Sepal.Width</code> is the most important of all the variables!</p>
</section>
<section id="flawed-variable-selection" class="level2">
<h2 class="anchored" data-anchor-id="flawed-variable-selection">Flawed variable selection</h2>
<p>The plot below shows that the more correlated variables we have, the lower the importance of ALL OF THEM AT THE SAME TIME! It’s as if the collinear variables split the importance among themselves.</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb5" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb5-1"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># fits a random forest on datasets with 1 to 20 repetitions of `Sepal.Length`</span></span>
<span id="cb5-2">rfs <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">map</span>(iris3, <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">~</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">randomForest</span>(Petal.Length <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">~</span> ., <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">data =</span> .x) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span> importance)</span>
<span id="cb5-3"></span>
<span id="cb5-4"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># extracts the importances of the `Sepal.Length` repetitions</span></span>
<span id="cb5-5">importancia <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">map_dfr</span>(rfs, <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">~</span>{</span>
<span id="cb5-6">  .x <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span> </span>
<span id="cb5-7">    <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">as.data.frame</span>() <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span> </span>
<span id="cb5-8">    tibble<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">::</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">rownames_to_column</span>() <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span> </span>
<span id="cb5-9">    dplyr<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">::</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">filter</span>(stringr<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">::</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">str_detect</span>(rowname, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"^Sepal.Length"</span>))</span>
<span id="cb5-10">}, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">.id =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"n_repeticoes"</span>) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb5-11">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">mutate</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">n_repeticoes =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">as.numeric</span>(n_repeticoes))</span>
<span id="cb5-12"></span>
<span id="cb5-13"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Plot of the number of multicollinear variables vs importance</span></span>
<span id="cb5-14">importancia <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb5-15">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">ggplot</span>(<span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">aes</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">x =</span> n_repeticoes, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">y =</span> IncNodePurity)) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span></span>
<span id="cb5-16">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">geom_point</span>() <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span></span>
<span id="cb5-17">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">geom_hline</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">yintercept =</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">40</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">size =</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">linetype =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"dashed"</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">colour =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"red"</span>) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span></span>
<span id="cb5-18">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">labs</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">x =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Qtd de repetições da coluna `Sepal.Length`"</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">y =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Importância"</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">title =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Gráfico da relação entre o número de variáveis multicolineares vs importância"</span>)</span></code></pre></div></div>
<div class="cell-output-display">
<div>
<figure class="figure">
<p><img src="https://athospd.github.io/athosdamiani/posts/2018-05-22-arvore-e-multicolinearidade/index_files/figure-html/unnamed-chunk-6-1.png" class="img-fluid figure-img" width="672"></p>
</figure>
</div>
</div>
</div>
<p>In practice, if we set an importance cutoff to drop variables (as illustrated by the red line), we would have a potential problem: we could be throwing away very important information.</p>
</section>
<section id="so-how-do-we-deal-with-multicollinearity" class="level2">
<h2 class="anchored" data-anchor-id="so-how-do-we-deal-with-multicollinearity">So how do we deal with multicollinearity?</h2>
<p>Some ways to deal with multicollinearity are:</p>
<ul>
<li>Looking at the correlation matrix</li>
<li><a href="http://www.sthda.com/english/articles/39-regression-model-diagnostics/160-multicollinearity-essentials-and-vif-in-r/">VIF</a></li>
<li><a href="https://machinelearningmastery.com/feature-selection-with-the-caret-r-package/">Recursive feature elimination</a></li>
</ul>
</section>
<section id="conclusion" class="level2">
<h2 class="anchored" data-anchor-id="conclusion">Conclusion</h2>
<p>Be careful when throwing everything into the pot! We should always worry about multicollinearity, even when fitting tree-based models.</p>
<p>Cheers!</p>


</section>

 ]]></description>
  <category>concepts</category>
  <category>machine learning</category>
  <category>random forest</category>
  <category>multicollinearity</category>
  <guid>https://athospd.github.io/athosdamiani/posts/2018-05-22-arvore-e-multicolinearidade/</guid>
  <pubDate>Tue, 22 May 2018 00:00:00 GMT</pubDate>
  <media:content url="https://athospd.github.io/athosdamiani/posts/2018-05-22-arvore-e-multicolinearidade/banner-arvore-e-multicolinearidade.webp" medium="image" type="image/webp"/>
</item>
<item>
  <title>Bring it on - Guess The Correlation Kaggle Competition</title>
  <dc:creator>Athos Damiani</dc:creator>
  <link>https://athospd.github.io/athosdamiani/posts/2018-03-29-guess-the-correlation/</link>
  <description><![CDATA[ 





<p>Hellooo community</p>
<p>We launched <a href="https://www.kaggle.com/c/guess-the-correlation/">a Kaggle competition</a> and now it’s time to show you are a DATA SCIENCE Jedi! =D</p>
<p>Link: <a href="https://www.kaggle.com/c/guess-the-correlation/">https://www.kaggle.com/c/guess-the-correlation/</a></p>
<p>We did this for sport, please don’t try to find any use for this application =P.</p>
<section id="the-game" class="level2">
<h2 class="anchored" data-anchor-id="the-game">The Game</h2>
<p>The <a href="http://guessthecorrelation.com/">Guess The Correlation</a> website puts a human face to face with a scatterplot and then challenges them to guess its linear correlation.</p>
<p>Our Kaggle challenge is built the same way. For example: what would you guess is the correlation of the data in the figure below?</p>
<center>
<img src="https://athospd.github.io/athosdamiani/posts/2018-03-29-guess-the-correlation/exemplo_img.webp" class="img-fluid">
</center>
<p>We generated 200 thousand png images like the one above, and each one has its correlation labeled so we can train a little model.</p>
</section>
<section id="goal" class="level2">
<h2 class="anchored" data-anchor-id="goal">Goal</h2>
<p>The goal is simple and direct: build a robot that computes the (linear) correlation just by looking at the scatterplot.</p>
<p>In <em>machine learnese</em>, we want</p>
<center>
<img src="https://athospd.github.io/athosdamiani/posts/2018-03-29-guess-the-correlation/f_img_92.webp" class="img-fluid">
</center>
<p>where this <strong>f</strong> is worthy of being the <strong>f</strong> in <strong>F@!#</strong>.</p>
</section>
<section id="initial-guess" class="level2">
<h2 class="anchored" data-anchor-id="initial-guess">Initial guess</h2>
<p><a href="https://github.com/jtrecenti">Julião</a> has already been working on this problem and took an initial shot at the code for you to get inspired. Well, “initial” is a stretch, because he went straight to a CNN with the help of the <a href="https://github.com/decryptr/decryptr">decryptr</a> package:</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb1" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb1-1"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">library</span>(keras)</span>
<span id="cb1-2"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">library</span>(tidyverse)</span>
<span id="cb1-3"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">library</span>(decryptr)</span>
<span id="cb1-4"></span>
<span id="cb1-5">path <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"."</span></span>
<span id="cb1-6">arqs <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">dir</span>(<span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">paste0</span>(path,  <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"/train_imgs"</span>), <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">full.names =</span> <span class="cn" style="color: #8f5902;
background-color: null;
font-style: inherit;">TRUE</span>)</span>
<span id="cb1-7">resp <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> readr<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">::</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">read_csv</span>(<span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">paste0</span>(path,  <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"/train_responses.csv"</span>))</span>
<span id="cb1-8"></span>
<span id="cb1-9">i_train <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">sample</span>(<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">:</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">nrow</span>(resp), <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">10000</span>)</span>
<span id="cb1-10">arqs_train <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> arqs[i_train]</span>
<span id="cb1-11">arqs_test <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> arqs[<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">-</span>i_train]</span>
<span id="cb1-12"></span>
<span id="cb1-13">gen <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">function</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">batch_size =</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">100</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">arqs =</span> arqs_train) {</span>
<span id="cb1-14">  f <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">function</span>() {</span>
<span id="cb1-15">    a <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">sort</span>(<span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">sample</span>(arqs, batch_size))</span>
<span id="cb1-16">    imgs <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> decryptr<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">::</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">read_captcha</span>(a)</span>
<span id="cb1-17">    captchas_t <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> purrr<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">::</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">transpose</span>(imgs)</span>
<span id="cb1-18">    xs <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> captchas_t<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span>x</span>
<span id="cb1-19">    xs <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> abind<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">::</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">abind</span>(xs, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">along =</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0.1</span>)</span>
<span id="cb1-20">    a_clean <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> a <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span> </span>
<span id="cb1-21">      <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">basename</span>() <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span> </span>
<span id="cb1-22">      tools<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">::</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">file_path_sans_ext</span>() <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span> </span>
<span id="cb1-23">      <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">tolower</span>()</span>
<span id="cb1-24">    corr <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> resp <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span> </span>
<span id="cb1-25">      <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">filter</span>(id <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%in%</span> a_clean) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span> </span>
<span id="cb1-26">      <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">arrange</span>(id) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span> </span>
<span id="cb1-27">      <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">pull</span>(corr)</span>
<span id="cb1-28">    data <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">list</span>(xs, corr)</span>
<span id="cb1-29">    data</span>
<span id="cb1-30">  }</span>
<span id="cb1-31">  f</span>
<span id="cb1-32">}</span>
<span id="cb1-33"></span>
<span id="cb1-34"></span>
<span id="cb1-35"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Create model</span></span>
<span id="cb1-36">model <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">keras_model_sequential</span>()</span>
<span id="cb1-37">model <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb1-38">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">layer_conv_2d</span>(</span>
<span id="cb1-39">    <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">input_shape =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">c</span>(<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">150</span>, <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">150</span>, <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span>),</span>
<span id="cb1-40">    <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">filters =</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">16</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">kernel_size =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">c</span>(<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">3</span>, <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">3</span>),</span>
<span id="cb1-41">    <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">padding =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"same"</span>,</span>
<span id="cb1-42">    <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">activation =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"relu"</span>) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb1-43">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">layer_max_pooling_2d</span>() <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb1-44">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">layer_conv_2d</span>(</span>
<span id="cb1-45">    <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">filters =</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">32</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">kernel_size =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">c</span>(<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">3</span>, <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">3</span>),</span>
<span id="cb1-46">    <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">padding =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"same"</span>,</span>
<span id="cb1-47">    <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">activation =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"relu"</span>) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb1-48">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">layer_max_pooling_2d</span>() <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb1-49">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">layer_conv_2d</span>(</span>
<span id="cb1-50">    <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">filters =</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">64</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">kernel_size =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">c</span>(<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">3</span>, <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">3</span>),</span>
<span id="cb1-51">    <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">padding =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"same"</span>,</span>
<span id="cb1-52">    <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">activation =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"relu"</span>) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb1-53">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">layer_max_pooling_2d</span>() <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb1-54">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">layer_flatten</span>() <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb1-55">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">layer_dense</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">units =</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">300</span>) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb1-56">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">layer_dropout</span>(.<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span>) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb1-57">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">layer_dense</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">units =</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">activation =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"tanh"</span>)</span>
<span id="cb1-58"></span>
<span id="cb1-59"></span>
<span id="cb1-60"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Compile and fit model</span></span>
<span id="cb1-61">model <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb1-62">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">compile</span>(</span>
<span id="cb1-63">    <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">optimizer =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"sgd"</span>,</span>
<span id="cb1-64">    <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">loss =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"mean_squared_error"</span>,</span>
<span id="cb1-65">    <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">metrics =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"mean_squared_error"</span>)</span>
<span id="cb1-66"></span>
<span id="cb1-67">model <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb1-68">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">fit_generator</span>(</span>
<span id="cb1-69">    <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">gen</span>(<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">100</span>, arqs_train),</span>
<span id="cb1-70">    <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">steps_per_epoch =</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">100</span>,</span>
<span id="cb1-71">    <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">validation_data =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">gen</span>(<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">100</span>, arqs_test),</span>
<span id="cb1-72">    <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">validation_steps =</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">10</span></span>
<span id="cb1-73">  )</span></code></pre></div></div>
</div>
<p>So? Can you get it 100% right? Or is it impossible?</p>
<p>Happy R’ing!</p>


</section>

 ]]></description>
  <category>announcements</category>
  <category>kaggle</category>
  <category>deep learning</category>
  <guid>https://athospd.github.io/athosdamiani/posts/2018-03-29-guess-the-correlation/</guid>
  <pubDate>Thu, 29 Mar 2018 00:00:00 GMT</pubDate>
  <media:content url="https://athospd.github.io/athosdamiani/posts/2018-03-29-guess-the-correlation/guess-the-correlation.webp" medium="image" type="image/webp"/>
</item>
<item>
  <title>How to make R tell you on telegram that bitcoin is cheap</title>
  <dc:creator>Athos Damiani</dc:creator>
  <link>https://athospd.github.io/athosdamiani/posts/2017-08-19-r-telegram-bitcoin/</link>
  <description><![CDATA[ 





<p>The highlight of the story is R sending a message to Telegram. Just to get you motivated, I’ll tell you about something fun I did with telegram and bitcoin prices.</p>
<div class="cell" data-layout-align="center">
<div class="cell-output-display">
<div class="quarto-figure quarto-figure-center">
<figure class="figure">
<p><img src="https://athospd.github.io/athosdamiani/posts/2017-08-19-r-telegram-bitcoin/r_telegram.gif" class="img-fluid quarto-figure quarto-figure-center figure-img"></p>
</figure>
</div>
</div>
</div>
<p>Hero of the day: <a href="https://github.com/lbraglia">Luca Biglia</a>, author of the R <code>telegram</code> package.</p>
<section id="what-well-do" class="level1">
<h1>What we’ll do</h1>
<p>As I mentioned, I’ll build a <strong>bitcoin tracker</strong> to illustrate, and by the end of the post we’ll have made R send a telegram when the price of bitcoin hits a given level.</p>
<div class="cell" data-layout-align="center">
<div class="cell-output-display">
<div class="quarto-figure quarto-figure-center">
<figure class="figure">
<p><img src="https://athospd.github.io/athosdamiani/posts/2017-08-19-r-telegram-bitcoin/serie_bitcoin.webp" class="img-fluid quarto-figure quarto-figure-center figure-img"></p>
</figure>
</div>
</div>
</div>
<section id="ingredients" class="level2">
<h2 class="anchored" data-anchor-id="ingredients">Ingredients</h2>
<p>For today’s craft project we’ll need:</p>
<ul>
<li>A <a href="https://core.telegram.org/bots">telegram bot</a></li>
<li>The R package <code>library(telegram)</code></li>
<li>The <a href="https://blinktrade.com/docs/">BlinkTrade</a> API</li>
<li>String and a tube of glue</li>
</ul>
<p>The steps we need to follow to get there are:</p>
<ol type="1">
<li>Create an .Rproj (RStudio project).</li>
<li>install and load the R <code>telegram</code> package.</li>
<li>Create a telegram bot with the help of <a href="https://telegram.me/botfather">BotFather</a>.</li>
<li>Connect R to the bot.</li>
<li>Query and store bitcoin prices every 30 seconds.</li>
<li>create an infinite loop in R to track the prices nonstop.</li>
<li>Make R send us a message when the price of bitcoin is above X, for example.</li>
<li>(extra) Come up with wild ideas of what to do with telegram + R.</li>
</ol>
</section>
<section id="introduction-to-the-components" class="level2">
<h2 class="anchored" data-anchor-id="introduction-to-the-components">Introduction to the components</h2>
<p>First of all, a brief introduction to the things that will show up here:</p>
<ul>
<li><a href="https://telegram.org/">telegram</a>: just like Whatsapp, but better.</li>
<li><a href="https://telegram.me/botfather">BotFather</a>: a contact on your phone (on telegram) that helps you create a telegram bot.</li>
<li><a href="https://en.wikipedia.org/wiki/Bitcoin">bitcoin</a>: a successful digital currency.</li>
<li><a href="https://blinktrade.com/docs/">BlinkTrade</a>: among other things, provides an API for real-time bitcoin values.</li>
</ul>
</section>
<section id="bitcoin-tracker" class="level2">
<h2 class="anchored" data-anchor-id="bitcoin-tracker">Bitcoin tracker</h2>
<p>Go ahead and create your .Rproj, run <code>install.packages("telegram");library(telegram)</code> and let’s go straight to the third step.</p>
<section id="step-3-telegram-bot---create-a-bot" class="level3">
<h3 class="anchored" data-anchor-id="step-3-telegram-bot---create-a-bot">Step 3: Telegram bot - Create a bot</h3>
<p>A bot is like one more person in your telegram contact list, except they are machines instead of humans and respond to specific commands. The fun part is that you can customize these commands any way you like! You just need a purpose and to know how to program it.</p>
<p>The README on the <a href="https://github.com/lbraglia/telegram">package’s github</a> is very good! I’ll summarize what’s there with small changes:</p>
<ol type="1">
<li>Go to your telegram and search for BotFather as if you were looking for a person in your contact list. Start a chat with him!</li>
<li>Send the text “/start” and then “/newbot”. Give your bot a name (it can be a cute name) and then a username for your bot, which must end in <em>bot</em>.</li>
<li>Now copy the token BotFather sent you and paste it into <code>.Renviron</code>. My bot has the username <code>AthosDamianiBot</code>, so I should name the variable like this:</li>
</ol>
<div class="cell" data-layout-align="center">
<div class="cell-output-display">
<div class="quarto-figure quarto-figure-center">
<figure class="figure">
<p><img src="https://athospd.github.io/athosdamiani/posts/2017-08-19-r-telegram-bitcoin/Renvirom_1.webp" class="img-fluid quarto-figure quarto-figure-center figure-img"></p>
</figure>
</div>
</div>
</div>
<p>If you use this naming convention you can use the <code>bot_token()</code> function to get your token. Otherwise you’ll have to resort to <code>Sys.getenv()</code>.</p>
<p><strong>NOTE:</strong> Restart R so that <code>.Renviron</code> takes effect.</p>
</section>
<section id="step-4-telegram-bot---connect-to-r" class="level3">
<h3 class="anchored" data-anchor-id="step-4-telegram-bot---connect-to-r">Step 4: Telegram bot - Connect to R</h3>
<ol type="1">
<li>Load the package and create a <code>TGBot</code> object for the bot you created earlier:</li>
</ol>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb1" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb1-1"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">library</span>(telegram)</span>
<span id="cb1-2"></span>
<span id="cb1-3">bot <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> TGBot<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">new</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">token =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">bot_token</span>(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">'AthosDamianiBot'</span>))</span>
<span id="cb1-4"></span>
<span id="cb1-5">bot<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">getMe</span>()</span></code></pre></div></div>
</div>
<ol start="2" type="1">
<li><p>Now we need the <code>chat_id</code>. To get it, search for your bot on telegram as if it were a contact (just like you did with BotFather) and start a chat with it.</p></li>
<li><p>In R, call the <code>bot$getUpdates()</code> method to get the messages you sent it into R and, finally, find the <code>chat_id</code> hidden in <code>msgs</code>.</p></li>
</ol>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb2" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb2-1">msgs <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> bot<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">getUpdates</span>()</span>
<span id="cb2-2"></span>
<span id="cb2-3">msgs<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span>message<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span>chat<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span>id[<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span>]</span>
<span id="cb2-4"></span>
<span id="cb2-5">[<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span>] <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">135717340</span></span></code></pre></div></div>
</div>
<ol start="4" type="1">
<li>With the <code>chat_id</code> in hand, set it as the default <code>chat_id</code>.</li>
</ol>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb3" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb3-1">bot<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">set_default_chat_id</span>(<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">135717340</span>)</span></code></pre></div></div>
</div>
<p>At this point we’re ready to interact with our bot!</p>
</section>
<section id="step-5-query-bitcoin-prices" class="level3">
<h3 class="anchored" data-anchor-id="step-5-query-bitcoin-prices">Step 5: Query bitcoin prices</h3>
<p>Bitcoin prices are provided by the <a href="https://blinktrade.com/docs/">BlinkTrade</a> API, which is very simple to use: just grab the json that the url in the code below returns. While I’m at it, I give it a little polish to make it a data.frame with the query date attached.</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb4" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb4-1"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">library</span>(jsonlite)</span>
<span id="cb4-2"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">library</span>(tidyverse)</span>
<span id="cb4-3"></span>
<span id="cb4-4">safe_fromJSON <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">safely</span>(fromJSON, <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">as.numeric</span>(<span class="cn" style="color: #8f5902;
background-color: null;
font-style: inherit;">NA</span>))</span>
<span id="cb4-5">nova_consulta_list <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">safe_fromJSON</span>(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"https://api.blinktrade.com/api/v1/BRL/ticker?crypto_currency=BTC"</span>)</span>
<span id="cb4-6"></span>
<span id="cb4-7">nova_consulta <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> nova_consulta_list<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span>result <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb4-8">        as.tibble <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb4-9">        <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">mutate</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">timestamp =</span> lubridate<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">::</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">now</span>())</span>
<span id="cb4-10"></span>
<span id="cb4-11">nova_consulta</span>
<span id="cb4-12"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># A tibble: 1 x 9</span></span>
<span id="cb4-13">      high      vol      buy  last      low   pair  sell vol_brl           timestamp</span>
<span id="cb4-14">     <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;</span>dbl<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&gt;</span>    <span class="er" style="color: #AD0000;
background-color: null;
font-style: inherit;">&lt;</span>dbl<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&gt;</span>    <span class="er" style="color: #AD0000;
background-color: null;
font-style: inherit;">&lt;</span>dbl<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&gt;</span> <span class="er" style="color: #AD0000;
background-color: null;
font-style: inherit;">&lt;</span>dbl<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&gt;</span>    <span class="er" style="color: #AD0000;
background-color: null;
font-style: inherit;">&lt;</span>dbl<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&gt;</span>  <span class="er" style="color: #AD0000;
background-color: null;
font-style: inherit;">&lt;</span>chr<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&gt;</span> <span class="er" style="color: #AD0000;
background-color: null;
font-style: inherit;">&lt;</span>dbl<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&gt;</span>   <span class="er" style="color: #AD0000;
background-color: null;
font-style: inherit;">&lt;</span>dbl<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&gt;</span>              <span class="er" style="color: #AD0000;
background-color: null;
font-style: inherit;">&lt;</span>dttm<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&gt;</span></span>
<span id="cb4-15"><span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">14438.21</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">511.5911</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">14077.01</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">14200</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">13801.04</span> BTCBRL <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">14200</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">7257317</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">2017-08-18</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">17</span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">:</span><span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">09</span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">:</span><span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">44</span></span></code></pre></div></div>
</div>
<p><strong>NOTE:</strong> I used the <code>safely()</code> adverb because the API can choke at any moment, which would make R return an error that would stop the price tracking.</p>
</section>
<section id="step-6-infinite-loop-to-track-the-prices" class="level3">
<h3 class="anchored" data-anchor-id="step-6-infinite-loop-to-track-the-prices">Step 6: Infinite loop to track the prices</h3>
<p>The skeleton of the tracker is made of an infinite loop (<code>while(TRUE)</code>), a <code>historico.RData</code> data.frame, a wait time between one query and the next (30 seconds by default) and the query itself.</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb5" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb5-1"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># initializes historico.RData</span></span>
<span id="cb5-2"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># historico &lt;- nova_consulta</span></span>
<span id="cb5-3"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># save(historico, file = "historico.RData")</span></span>
<span id="cb5-4"></span>
<span id="cb5-5">acompanhar_bitcoin <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">function</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">frequencia =</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">30</span>) {</span>
<span id="cb5-6">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">load</span>(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"historico.RData"</span>)</span>
<span id="cb5-7"></span>
<span id="cb5-8">  <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># infinite loop</span></span>
<span id="cb5-9">  <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">while</span>(<span class="cn" style="color: #8f5902;
background-color: null;
font-style: inherit;">TRUE</span>) {</span>
<span id="cb5-10">    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># gets the brazilian bitcoin quote (BTCBRL) from the blinktrade API</span></span>
<span id="cb5-11">    nova_consulta_list <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">safe_fromJSON</span>(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"https://api.blinktrade.com/api/v1/BRL/ticker?crypto_currency=BTC"</span>)</span>
<span id="cb5-12"></span>
<span id="cb5-13">    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># checks whether the API returned a list</span></span>
<span id="cb5-14">    <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">if</span>(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"list"</span> <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%in%</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">class</span>(nova_consulta_list<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span>result)) {</span>
<span id="cb5-15">      nova_consulta <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> nova_consulta_list<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span>result <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb5-16">        as.tibble <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb5-17">        <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">mutate</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">timestamp =</span> lubridate<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">::</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">now</span>())</span>
<span id="cb5-18"></span>
<span id="cb5-19">      <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># ---------------------</span></span>
<span id="cb5-20">      <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#</span></span>
<span id="cb5-21">      <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># space reserved for the rules!</span></span>
<span id="cb5-22">      <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#</span></span>
<span id="cb5-23">      <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># ---------------------</span></span>
<span id="cb5-24"></span>
<span id="cb5-25">      <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># stores the query</span></span>
<span id="cb5-26">      historico <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">bind_rows</span>(historico, nova_consulta)</span>
<span id="cb5-27">      <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">save</span>(historico, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">file =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"historico.RData"</span>)</span>
<span id="cb5-28">    }</span>
<span id="cb5-29">  }</span>
<span id="cb5-30">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">Sys.sleep</span>(frequencia)</span>
<span id="cb5-31">}</span></code></pre></div></div>
</div>
</section>
<section id="step-7-rules-for-telegram-messages" class="level3">
<h3 class="anchored" data-anchor-id="step-7-rules-for-telegram-messages">Step 7: Rules for telegram messages</h3>
<p>Now it’s time to decide what the bot should warn us about! I left two simple examples below using the <code>bot$sendMessage()</code> method, which, as the name suggests, makes the bot send us a message. Now, every time the price of bitcoin is below R$13,600 or above R$14,600 I’ll know right away!</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb6" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb6-1"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># if the quote value meets some criterion, sends a message via telegram.</span></span>
<span id="cb6-2"><span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">if</span>(nova_consulta<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span>buy <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">13600</span> <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&amp;</span> nova_consulta<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span>last <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&gt;</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">13900</span>) {</span>
<span id="cb6-3">  bot<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">sendMessage</span>(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">'baixa!'</span>)</span>
<span id="cb6-4">  bot<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">sendMessage</span>(nova_consulta<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span>buy)</span>
<span id="cb6-5">}</span>
<span id="cb6-6"></span>
<span id="cb6-7"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># ...</span></span>
<span id="cb6-8"></span>
<span id="cb6-9"><span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">if</span>(nova_consulta<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span>buy <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&gt;</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">14600</span> <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&amp;</span> nova_consulta<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span>last <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">14500</span>) {</span>
<span id="cb6-10">  bot<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">sendMessage</span>(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">'alta!'</span>)</span>
<span id="cb6-11">  bot<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">sendMessage</span>(nova_consulta<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span>buy)</span>
<span id="cb6-12">}</span></code></pre></div></div>
</div>
</section>
<section id="extra-step-wild-ideas-of-what-to-do-with-telegram-r" class="level3">
<h3 class="anchored" data-anchor-id="extra-step-wild-ideas-of-what-to-do-with-telegram-r">Extra Step: Wild ideas of what to do with telegram + R</h3>
<p>Creativity rules here. You can send plots, sounds, stickers and any kind of media. I made a ggplot to keep an eye on the trend, so I could ask my bot to show me the plot at any time.</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb7" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb7-1"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># ... this goes inside the acompanhar_bitcoin() function</span></span>
<span id="cb7-2">p <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">ggplot</span>(historico <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb7-3">              <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">gather</span>(indicador, valor, high, low, buy, sell, last)) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span></span>
<span id="cb7-4">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">geom_line</span>(<span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">aes</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">x =</span> timestamp, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">y =</span> valor, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">colour =</span> indicador))</span>
<span id="cb7-5"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">print</span>(p)</span></code></pre></div></div>
</div>
<p>You can also program commands to pass to the Bot so it builds something for us, for example, you could create a command to buy or sell bitcoins in real time!</p>
<p>Julio once created a bot that was practically R itself: you sent code as a message and it returned R’s output! Later I’ll ask him to post something about it.</p>
<p>Anyway, there’s room to be clever here. All it takes is that brilliant shower idea.</p>
<p>Homework: - Send the “gemidão do zap” (a Brazilian prank audio) to 3 friends with a line of code.</p>
<p>cheers!</p>


</section>
</section>
</section>

 ]]></description>
  <category>tutorials</category>
  <category>telegram</category>
  <category>bitcoin</category>
  <guid>https://athospd.github.io/athosdamiani/posts/2017-08-19-r-telegram-bitcoin/</guid>
  <pubDate>Sat, 19 Aug 2017 00:00:00 GMT</pubDate>
  <media:content url="https://athospd.github.io/athosdamiani/posts/2017-08-19-r-telegram-bitcoin/r_telegram_bitcoin.webp" medium="image" type="image/webp"/>
</item>
<item>
  <title>Logistic Regression in: the world’s smallest deep learning</title>
  <dc:creator>Athos Damiani</dc:creator>
  <link>https://athospd.github.io/athosdamiani/posts/2017-07-29-segundo-menor-dl/</link>
  <description><![CDATA[ 





<section id="goals" class="level2">
<h2 class="anchored" data-anchor-id="goals">Goals</h2>
<p>The purpose of this post is to:</p>
<ul>
<li>learn how to fit a logistic regression with keras</li>
<li>learn how to do a PCA with keras</li>
<li>bring Deep Learning closer to what most data analysts already knew.</li>
<li>encourage everyone who came before deep learning to study it and get comfortable with what’s new around it.</li>
<li>show that many professionals in the machine learning field already knew a large part of what deep learning uses.</li>
<li>raise a discussion about some myths that are not constructive for the data analyst community.</li>
</ul>
</section>
<section id="motivation" class="level2">
<h2 class="anchored" data-anchor-id="motivation">Motivation</h2>
<p>I read statisticians, computer scientists, data engineers and the like questioning the future of Machine Learning and whether everything we knew before about statistical modeling had become obsolete (like this question on Quora: <a href="https://www.quora.com/Should-I-quit-machine-learning">Should I Quit Machine Learning?</a>).</p>
<p>And in conversations with people close to me I noticed some excitement about the novelty and frustration with the “obsolescence” of what they had spent time studying before.</p>
<p>To make it worse, opportunists rode this wave to do malicious marketing aimed at devaluing and dividing the data analyst community. Something very similar to what happened with other buzzwords like <em>data science</em>, <em>big data</em>, <em>Python versus R</em> and <em>machine learning</em> itself. Before, there was the classic advertising that company X used MACHINE LEARNING instead of predictive models. Now things have evolved and they resort to the word Deep Learning.</p>
<p>What really matters:</p>
<ol type="1">
<li>Deep Learning is big news and has put Artificial Intelligence in the spotlight.</li>
<li>Whoever knew Machine Learning before will be able to apply 95% of their knowledge in Deep Learning applications (including bayesianism, bootstrap, inference, probability and the whole lot).</li>
<li>Deep Learning has to be seen as one more tool in the data analyst’s toolbox, not a replacement.</li>
</ol>
<p>And to address this question I decided to fit a logistic regression using deep learning so that everyone who has ever fit a logistic regression can say they have built a neural network too! I confess a slightly provocative motivation, but what fun would it be otherwise? =P</p>
</section>
<section id="what-well-do" class="level2">
<h2 class="anchored" data-anchor-id="what-well-do">What we’ll do</h2>
<ul>
<li>Logistic regression for <img src="https://latex.codecogs.com/png.latex?Y_1"> (with <code>glm</code>)</li>
<li>Deep Learning for <img src="https://latex.codecogs.com/png.latex?Y_1"> (with <code>keras</code>)</li>
<li>Show that logistic regression is not the best for <img src="https://latex.codecogs.com/png.latex?Y_2"> and that Deep Learning goes beyond the limitation of linear models (with <code>glm</code>)</li>
<li>Deep Learning for <img src="https://latex.codecogs.com/png.latex?Y_2"> (with <code>keras</code>)</li>
</ul>
<p>Let’s get to work.</p>
<section id="packages" class="level3">
<h3 class="anchored" data-anchor-id="packages">Packages</h3>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb1" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb1-1"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">library</span>(keras)</span>
<span id="cb1-2"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">library</span>(dplyr)</span>
<span id="cb1-3"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">library</span>(tidyr)</span>
<span id="cb1-4"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">library</span>(forcats)</span>
<span id="cb1-5"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">library</span>(ggplot2)</span></code></pre></div></div>
</div>
</section>
</section>
<section id="logistic-regression-versus-deep-learning" class="level2">
<h2 class="anchored" data-anchor-id="logistic-regression-versus-deep-learning">Logistic regression versus Deep Learning</h2>
<p>Time to fit models to the same data in two different ways: logistic regression with <code>glm</code> and deep learning with <code>keras</code>.</p>
<section id="simulated-data" class="level3">
<h3 class="anchored" data-anchor-id="simulated-data">Simulated data</h3>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb2" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb2-1">logit <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">function</span>(p) <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">log</span>(p) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">log</span>(<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span> <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">-</span> p)</span>
<span id="cb2-2">logistic <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">function</span>(x) <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">/</span>(<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span> <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">exp</span>(<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">-</span>x))</span>
<span id="cb2-3"></span>
<span id="cb2-4">n <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">100000</span></span>
<span id="cb2-5"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">set.seed</span>(<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">19880923</span>)</span>
<span id="cb2-6">df <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">data_frame</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">x =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">runif</span>(n, <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">-</span><span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">2</span>, <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">2.5</span>)) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb2-7">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">mutate</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">y_1 =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">rbinom</span>(n, <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">prob =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">logistic</span>(<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">-</span><span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span> <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">2</span> <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">*</span> x)), <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># y_1</span></span>
<span id="cb2-8">         <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">y_2 =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">rbinom</span>(n, <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">prob =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">logistic</span>(<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">-</span><span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span> <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">2</span> <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">*</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">tanh</span>(<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">-</span><span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span> <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">2</span> <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">*</span> x)))) <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># y_2</span></span></code></pre></div></div>
</div>
<p>The code above created two response variables (<em>targets</em>). In mathematical notation, they are defined as follows:</p>
<p><strong>Response <code>y_1</code></strong></p>
<p><img src="https://latex.codecogs.com/png.latex?E%5BY_1%7Cx%5D%20=%20%5Ctext%7Blogistic%7D%7B(-1%20+%202x)%7D%20=%20%5Cfrac%7B1%7D%7B1%20+%20e%5E%7B%7B-(-1%20+%202x)%7D%7D%7D"></p>
<p><strong>Response <code>y_2</code></strong></p>
<p><img src="https://latex.codecogs.com/png.latex?E%5BY_2%7Cx%5D%20=%20%5Ctext%7Blogistic%7D%7B(-1%20+%202%5Ctanh(-1%20+%202x))%7D%20=%20%5Cfrac%7B1%7D%7B1%20+%20e%5E%7B%7B-(-1%20+%202%5Ctanh(-1%20+%202x))%7D%7D%7D"></p>
<p><img src="https://latex.codecogs.com/png.latex?x"> is linear in the logit of <code>y_1</code>, so logistic regression will do well at finding the parameters <img src="https://latex.codecogs.com/png.latex?-1"> and <img src="https://latex.codecogs.com/png.latex?2">. However, <img src="https://latex.codecogs.com/png.latex?x"> is not linear in the logit of <code>y_2</code>, so logistic regression won’t be able to faithfully represent the generator of <code>y_2</code>.</p>
<p><strong>NOTE 1:</strong> The form <img src="https://latex.codecogs.com/png.latex?%5Ctext%7Blogistic%7D%7B(%5Cbeta_0%20+%20%5Cbeta_1%5Ctanh(%5Cbeta_2%20+%20%5Cbeta_3X))%7D"> has parameters inside the <code>tanh</code> function, which means our hypothesis for <img src="https://latex.codecogs.com/png.latex?E%5BY_2%7Cx%5D"> is no longer <strong>linear</strong> in the parameters. That’s why linear models (as the name suggests) are no longer suitable. And <strong>non-linearity</strong> is one of the generalizations neural networks give us! (yes, this is very relevant)</p>
<p><strong>NOTE 2:</strong> of course, in this very simple one-variable case we could inspect the data to find good transformations of <img src="https://latex.codecogs.com/png.latex?x"> so that the logistic fit would be as good as a neural network’s, but if we added many other variables things would get complicated!</p>
<p>In neural network notation, the formulas above look like this:</p>
<p><strong>Response <code>y_1</code></strong></p>
<div class="cell">
<div class="cell-output-display">
<div>
<figure class="figure">
<p><img src="https://athospd.github.io/athosdamiani/posts/2017-07-29-segundo-menor-dl/y1.webp" class="img-fluid figure-img"></p>
</figure>
</div>
</div>
</div>
<p><strong>Response <code>y_2</code></strong></p>
<div class="cell">
<div class="cell-output-display">
<div>
<figure class="figure">
<p><img src="https://athospd.github.io/athosdamiani/posts/2017-07-29-segundo-menor-dl/y2.webp" class="img-fluid figure-img"></p>
</figure>
</div>
</div>
</div>
<p>What was a <strong>link function</strong> in GLM became an <strong>activation function</strong> in neural networks (at the end I talk more about vocabulary that changed).</p>
</section>
<section id="a-look-at-the-data" class="level3">
<h3 class="anchored" data-anchor-id="a-look-at-the-data">A look at the data</h3>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb3" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb3-1"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># skimr::skim(df) %&gt;% skim_print %&gt;% with(numeric)  %&gt;% mutate_if(is.numeric, round, 2) %&gt;% DT::datatable()</span></span>
<span id="cb3-2"></span>
<span id="cb3-3">df <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb3-4">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">gather</span>(y_id, y_val, y_1, y_2) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb3-5">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">mutate</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">x_cat =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">cut_number</span>(x, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">n =</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">70</span>)) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb3-6">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">group_by</span>(x_cat, y_id) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb3-7">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">summarise</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">p =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">mean</span>(y_val),</span>
<span id="cb3-8">            <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">n =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">n</span>()) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb3-9">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">mutate</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">logit_p =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">logit</span>(p)) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb3-10">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">gather</span>(transformacao, p, p, logit_p) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb3-11">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">mutate</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">transformacao =</span> transformacao <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span> fct_inorder <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">fct_recode</span>(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"logit(p)"</span> <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">=</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"logit_p"</span>)) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb3-12">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">ggplot</span>() <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span></span>
<span id="cb3-13">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">geom_point</span>(<span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">aes</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">x =</span> x_cat, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">y =</span> p, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">colour =</span> y_id)) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span></span>
<span id="cb3-14">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">theme_minimal</span>(<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">20</span>) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span></span>
<span id="cb3-15">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">theme</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">axis.text.x =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">element_text</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">angle =</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">90</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">hjust =</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">vjust =</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0.5</span>)) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span></span>
<span id="cb3-16">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">facet_wrap</span>(<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">~</span>forcats<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">::</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">fct_inorder</span>(transformacao), <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">nrow =</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">scales =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"free_y"</span>) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span></span>
<span id="cb3-17">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">labs</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">x =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"x"</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">colour =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"resposta"</span>) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span></span>
<span id="cb3-18">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">theme</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">axis.text.x =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">element_blank</span>(),<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">panel.grid.major.x =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">element_blank</span>(), <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">panel.grid.minor.y =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">element_blank</span>())</span>
<span id="cb3-19"><span class="do" style="color: #5E5E5E;
background-color: null;
font-style: italic;">## `summarise()` has grouped output by 'x_cat'. You can override using the</span></span>
<span id="cb3-20"><span class="do" style="color: #5E5E5E;
background-color: null;
font-style: italic;">## `.groups` argument.</span></span></code></pre></div></div>
<div class="cell-output-display">
<div>
<figure class="figure">
<p><img src="https://athospd.github.io/athosdamiani/posts/2017-07-29-segundo-menor-dl/index_files/figure-html/unnamed-chunk-6-1.png" class="img-fluid figure-img" width="1152"></p>
</figure>
</div>
</div>
</div>
<p>The plot on the right shows that <code>x</code> is proportional to the logit of the probabilities of <code>y_1</code> (in red), as it should be since we built it that way. With <code>y_2</code> (in blue), it still looks like a sigmoid even after the transformation.</p>
</section>
<section id="fitting-the-models" class="level3">
<h3 class="anchored" data-anchor-id="fitting-the-models">Fitting the models</h3>
<section id="logistic-regression-for-y_1-with-glm" class="level4">
<h4 class="anchored" data-anchor-id="logistic-regression-for-y_1-with-glm">Logistic regression for <img src="https://latex.codecogs.com/png.latex?Y_1"> (with <code>glm</code>)</h4>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb4" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb4-1"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># glm model 1 -------------------------------------------------------</span></span>
<span id="cb4-2">modelo_glm_1 <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">glm</span>(y_1 <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">~</span> x, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">data =</span> df, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">family =</span> binomial)</span></code></pre></div></div>
</div>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb5" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb5-1"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># coefficients</span></span>
<span id="cb5-2"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">coef</span>(modelo_glm_1)</span>
<span id="cb5-3"></span>
<span id="cb5-4"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># (Intercept)           x</span></span>
<span id="cb5-5"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#   -1.018434    2.011647</span></span>
<span id="cb5-6"></span>
<span id="cb5-7"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># accuracy</span></span>
<span id="cb5-8">conf_matrix_glm_1 <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">table</span>(modelo_glm_1<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span>fitted.values <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&gt;</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0.5</span>, df<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span>y_1)</span>
<span id="cb5-9"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">sum</span>(<span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">diag</span>(conf_matrix_glm_1))<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">/</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">sum</span>(conf_matrix_glm_1)</span>
<span id="cb5-10"></span>
<span id="cb5-11"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># [1] 0.85061</span></span></code></pre></div></div>
</div>
<p>The estimates came out very close to the true values <img src="https://latex.codecogs.com/png.latex?%5Cbeta_0%20=%20-1"> and <img src="https://latex.codecogs.com/png.latex?%5Cbeta_1%20=%202">.</p>
<p>The accuracy was <strong>85%</strong>.</p>
</section>
<section id="deep-learning-for-y_1-with-keras" class="level4">
<h4 class="anchored" data-anchor-id="deep-learning-for-y_1-with-keras">Deep Learning for <img src="https://latex.codecogs.com/png.latex?Y_1"> (with <code>keras</code>)</h4>
<p>Let’s build our hypothesis for <img src="https://latex.codecogs.com/png.latex?E%5BY_1%7Cx%5D">.</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb6" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb6-1"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># keras model 1 --------------------------------------------------------</span></span>
<span id="cb6-2"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># input: 1 variable: x.</span></span>
<span id="cb6-3">input_keras_1 <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">layer_input</span>(<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">name =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"modelo_keras_1"</span>)</span>
<span id="cb6-4"></span>
<span id="cb6-5"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># output: there are no hidden layers, only the logit link function directly.</span></span>
<span id="cb6-6">output_keras_1 <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> input_keras_1 <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb6-7">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">layer_dense</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">units =</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">name =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"camada_unica"</span>) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb6-8">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">layer_activation</span>(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"sigmoid"</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">input_shape =</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">name =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"link_logistic"</span>) <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># sigmoid in tensorflow is the logistic</span></span>
<span id="cb6-9"></span>
<span id="cb6-10"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># keras_model is what builds our hypothesis f(x) (from E[y] = f(x))</span></span>
<span id="cb6-11">modelo_keras_1 <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">keras_model</span>(input_keras_1, output_keras_1)</span>
<span id="cb6-12"></span>
<span id="cb6-13"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#</span></span>
<span id="cb6-14"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">summary</span>(modelo_keras_1)</span></code></pre></div></div>
</div>
<pre><code>Model
_____________________________________________________________
Layer (type)                     Output Shape    Param #
=============================================================
modelo_keras_1 (InputLayer)      (None, 1)       0
_____________________________________________________________
camada_unica (Dense)             (None, 1)       2
_____________________________________________________________
link_logistic (Activation)       (None, 1)       0
=============================================================
Total params: 2
Trainable params: 2
Non-trainable params: 0
_____________________________________________________________</code></pre>
<p>The hypothesis we built has 2 parameters. Looks right! <img src="https://latex.codecogs.com/png.latex?%5Cbeta_0"> and <img src="https://latex.codecogs.com/png.latex?%5Cbeta_1">.</p>
<p><strong>Now it’s the loss function’s turn.</strong></p>
<p>Since our goal is to build a logistic regression, we’ll choose the <a href="http://deeplearning.net/software/theano/library/tensor/nnet/nnet.html#theano.tensor.nnet.nnet.binary_crossentropy">binary_crossentropy</a> loss function, which is a synonym for the logistic <a href="https://en.wikipedia.org/wiki/Deviance_(statistics)">deviance</a>, the more common term in the statistics world.</p>
<p>The <code>'accuracy'</code> metric doesn’t go into the loss function optimizer; we use it to compare the models we create. Here we’ll compare against the <code>glm</code> model fitted above (but, for example, with rare events <code>'accuracy'</code> won’t be very informative, so we could use <code>'auc'</code>, <code>'gini'</code>, etc.).</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb8" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb8-1">modelo_keras_1 <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">compile</span>(</span>
<span id="cb8-2">  <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">loss =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">'binary_crossentropy'</span>,</span>
<span id="cb8-3">  <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">optimizer =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">optimizer_sgd</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">lr =</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0.4</span>),</span>
<span id="cb8-4">  <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">metrics =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">c</span>(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">'accuracy'</span>)</span>
<span id="cb8-5">)</span>
<span id="cb8-6"></span>
<span id="cb8-7">modelo_keras_1_fit <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> modelo_keras_1 <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">fit</span>(</span>
<span id="cb8-8">  <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">x =</span> df<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span>x,</span>
<span id="cb8-9">  <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">y =</span> df<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span>y_1,</span>
<span id="cb8-10">  <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">epochs =</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">20</span>,</span>
<span id="cb8-11">  <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">batch_size =</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1000</span>,</span>
<span id="cb8-12">  <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">verbose =</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">0</span></span>
<span id="cb8-13">)</span></code></pre></div></div>
</div>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb9" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb9-1"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># coefficients</span></span>
<span id="cb9-2">modelo_keras_1 <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">get_layer</span>(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"camada_unica"</span>) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span> get_weights</span>
<span id="cb9-3"></span>
<span id="cb9-4"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># [[1]]</span></span>
<span id="cb9-5"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#          [,1]</span></span>
<span id="cb9-6"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># [1,] 2.000054</span></span>
<span id="cb9-7"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#</span></span>
<span id="cb9-8"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># [[2]]</span></span>
<span id="cb9-9"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># [1] -1.015561</span></span>
<span id="cb9-10"></span>
<span id="cb9-11"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># accuracy</span></span>
<span id="cb9-12">loss_and_metrics_1 <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> modelo_keras_1 <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">evaluate</span>(df<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span>x, df<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span>y_1, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">batch =</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">100000</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">verbose =</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">0</span>)</span>
<span id="cb9-13">loss_and_metrics_1[[<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">2</span>]]</span>
<span id="cb9-14"></span>
<span id="cb9-15"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># [1] 0.85053</span></span></code></pre></div></div>
</div>
<p>Identical results! It had to be so, because we built the same hypothesis and the same loss function as <code>glm</code>.</p>
</section>
<section id="logistic-regression-for-y_2-with-glm" class="level4">
<h4 class="anchored" data-anchor-id="logistic-regression-for-y_2-with-glm">Logistic regression for <img src="https://latex.codecogs.com/png.latex?Y_2"> (with <code>glm</code>)</h4>
<p>To model <img src="https://latex.codecogs.com/png.latex?Y_2"> we’ll step onto ground that linear models don’t tread. First I try to fit a curve using <code>x</code> and the transformation <code>tanh(x)</code>. Let’s suppose I chose this predictor after a thorough and lengthy inspection of the data (I tried to simulate roughly what I would do in a modeling task where I’d have to build the features by hand).</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb10" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb10-1"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># glm model 2 -------------------------------------------------------</span></span>
<span id="cb10-2">modelo_glm_2 <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">glm</span>(y_2 <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">~</span> x <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">tanh</span>(x), <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">data =</span> df, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">family =</span> binomial)</span>
<span id="cb10-3"></span>
<span id="cb10-4"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># coefficients</span></span>
<span id="cb10-5"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">coef</span>(modelo_glm_2)</span>
<span id="cb10-6"></span>
<span id="cb10-7"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># (Intercept)           x     tanh(x)</span></span>
<span id="cb10-8"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#  -1.6698641   0.3043212   2.0936353</span></span>
<span id="cb10-9"></span>
<span id="cb10-10"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># accuracy</span></span>
<span id="cb10-11">conf_matrix_glm_2 <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">table</span>(modelo_glm_2<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span>fitted.values <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&gt;</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0.5</span>, df<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span>y_2)</span>
<span id="cb10-12"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">sum</span>(<span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">diag</span>(conf_matrix_glm_2))<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">/</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">sum</span>(conf_matrix_glm_2)</span>
<span id="cb10-13"></span>
<span id="cb10-14"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># [1] 0.82204</span></span></code></pre></div></div>
</div>
<p>82% accuracy, not bad. But the hypothesis and parameters were different from the true data generator. Let’s use neural networks to solve the non-linearity problem.</p>
</section>
<section id="deep-learning-for-y_2-with-keras" class="level4">
<h4 class="anchored" data-anchor-id="deep-learning-for-y_2-with-keras">Deep Learning for <img src="https://latex.codecogs.com/png.latex?Y_2"> (with <code>keras</code>)</h4>
<p>Hypothesis for <img src="https://latex.codecogs.com/png.latex?E%5BY_2%7Cx%5D">.</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb11" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb11-1"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># keras model 2 --------------------------------------------------------</span></span>
<span id="cb11-2">input_keras_2 <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">layer_input</span>(<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">name =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"modelo_keras_2"</span>)</span>
<span id="cb11-3"></span>
<span id="cb11-4">output_keras_2 <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> input_keras_2 <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb11-5">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">layer_dense</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">units =</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">name =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"camada_um"</span>) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb11-6">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">layer_activation</span>(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"tanh"</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">input_shape =</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">name =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"tanh_de_dentro"</span>) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb11-7">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">layer_dense</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">units =</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">input_shape =</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">name =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"camada_dois"</span>) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb11-8">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">layer_activation</span>(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"sigmoid"</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">input_shape =</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">name =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"link_logistic"</span>)</span>
<span id="cb11-9"></span>
<span id="cb11-10">modelo_keras_2 <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">keras_model</span>(input_keras_2, output_keras_2)</span>
<span id="cb11-11"></span>
<span id="cb11-12"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">summary</span>(modelo_keras_2)</span></code></pre></div></div>
</div>
<pre><code>Model
_____________________________________________________________
Layer (type)                Output Shape           Param #
=============================================================
modelo_keras_2 (InputLayer) (None, 1)              0
_____________________________________________________________
camada_um (Dense)           (None, 1)              2
_____________________________________________________________
tanh_de_dentro (Activation) (None, 1)              0
_____________________________________________________________
camada_dois (Dense)         (None, 1)              2
_____________________________________________________________
link_logistic (Activation)  (None, 1)              0
=============================================================
Total params: 4.0
Trainable params: 4.0
Non-trainable params: 0.0
_____________________________________________________________
</code></pre>
<p>Four ‘trainable’ parameters, that’s it! Two parameters inside the <code>tanh</code> and the two parameters outside. We need keras to give us back -1, 2, -1 and 2, the way we generated the data.</p>
<p><strong>Cost function</strong></p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb13" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb13-1">modelo_keras_2 <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">compile</span>(</span>
<span id="cb13-2">  <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">loss =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">'binary_crossentropy'</span>,</span>
<span id="cb13-3">  <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">optimizer =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">optimizer_sgd</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">lr =</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0.1</span>),</span>
<span id="cb13-4">  <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">metrics =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">c</span>(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">'accuracy'</span>)</span>
<span id="cb13-5">)</span>
<span id="cb13-6"></span>
<span id="cb13-7">modelo_keras_2_fit <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> modelo_keras_2 <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">fit</span>(</span>
<span id="cb13-8">  <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">x =</span> df<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span>x,</span>
<span id="cb13-9">  <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">y =</span> df<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span>y_2,</span>
<span id="cb13-10">  <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">epochs =</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">20</span>,</span>
<span id="cb13-11">  <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">batch_size =</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">100</span>,</span>
<span id="cb13-12">  <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">verbose =</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">0</span></span>
<span id="cb13-13">)</span>
<span id="cb13-14"></span>
<span id="cb13-15"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># coefficients</span></span>
<span id="cb13-16">modelo_keras_2 <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">get_layer</span>(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"camada_um"</span>) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span> get_weights</span>
<span id="cb13-17"></span>
<span id="cb13-18"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># [[1]]</span></span>
<span id="cb13-19"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#          [,1]</span></span>
<span id="cb13-20"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># [1,] 2.012015</span></span>
<span id="cb13-21"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#</span></span>
<span id="cb13-22"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># [[2]]</span></span>
<span id="cb13-23"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># [1] -1.058052</span></span>
<span id="cb13-24"></span>
<span id="cb13-25">modelo_keras_2 <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">get_layer</span>(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"camada_dois"</span>) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span> get_weights</span>
<span id="cb13-26"></span>
<span id="cb13-27"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># [[1]]</span></span>
<span id="cb13-28"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#          [,1]</span></span>
<span id="cb13-29"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># [1,] 1.981977</span></span>
<span id="cb13-30"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#</span></span>
<span id="cb13-31"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># [[2]]</span></span>
<span id="cb13-32"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># [1] -1.006567</span></span>
<span id="cb13-33"></span>
<span id="cb13-34"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># accuracy</span></span>
<span id="cb13-35">loss_and_metrics_2 <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> modelo_keras_2 <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">evaluate</span>(df<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span>x, df<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span>y_2, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">batch_size =</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">100000</span>)</span>
<span id="cb13-36">loss_and_metrics_2[[<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">2</span>]]</span>
<span id="cb13-37"></span>
<span id="cb13-38"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># [1] 0.82221</span></span></code></pre></div></div>
</div>
<p><strong>82%</strong> accuracy as well, but now the parameters are very close to the ones that generated the data! We just saw a set of parameters being found even with a non-linear relationship between them and the mean.</p>
<p>The accuracy of the two models ended up about the same, but the plot of the hypotheses found (below) shows that the <code>glm</code> curve is worse than the <code>keras</code> curve.</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb14" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb14-1">df <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb14-2">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">select</span>(x, y_2) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb14-3">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">mutate</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">x_cat =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">cut_number</span>(x, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">n =</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">50</span>)) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb14-4">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">group_by</span>(x_cat) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb14-5">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">summarise</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">p =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">mean</span>(y_2),</span>
<span id="cb14-6">            <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">x =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">mean</span>(x),</span>
<span id="cb14-7">            <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">keras =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">logistic</span>(<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">-</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">1.006567</span> <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">1.981977</span> <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">*</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">tanh</span>(<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">-</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">1.058052</span> <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">2.012015</span> <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">*</span> x)),</span>
<span id="cb14-8">            <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">glm =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">logistic</span>(<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">-</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">1.6698641</span> <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0.3043212</span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">*</span>x <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">2.0936353</span> <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">*</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">tanh</span>(x)),</span>
<span id="cb14-9">            <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">n =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">n</span>()) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb14-10">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">mutate</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">logit_p =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">logit</span>(p)) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb14-11">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">gather</span>(Modelo, estimativa, keras, glm) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb14-12">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">ggplot</span>() <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span></span>
<span id="cb14-13">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">geom_point</span>(<span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">aes</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">x =</span> x_cat, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">y =</span> p)) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span></span>
<span id="cb14-14">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">geom_line</span>(<span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">aes</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">x =</span> x_cat, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">y =</span> estimativa, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">colour =</span> Modelo, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">group =</span> Modelo)) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span></span>
<span id="cb14-15">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">theme_minimal</span>() <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span></span>
<span id="cb14-16">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">theme</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">axis.text.x =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">element_text</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">angle =</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">90</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">hjust =</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">vjust =</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0.5</span>)) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span></span>
<span id="cb14-17">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">labs</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">x =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"x"</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">colour =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"resposta"</span>)</span></code></pre></div></div>
<div class="cell-output-display">
<div>
<figure class="figure">
<p><img src="https://athospd.github.io/athosdamiani/posts/2017-07-29-segundo-menor-dl/index_files/figure-html/unnamed-chunk-15-1.png" class="img-fluid figure-img" width="672"></p>
</figure>
</div>
</div>
</div>
</section>
<section id="bonus-pca-with-an-autoencoder" class="level4">
<h4 class="anchored" data-anchor-id="bonus-pca-with-an-autoencoder">(Bonus) PCA with an autoencoder</h4>
<p>In practice, PCA and autoencoders are used to reduce the dimensionality of the data. PCA is a particular case of an autoencoder with only one layer and linear activation functions. The post <a href="http://curso-r.com/blog/2017/06/26/2017-06-26-construindo-autoencoders/">Construindo Autoencoders</a> (in Portuguese) shows how to build one and I recommend reading it.</p>
<p>Summary: the autoencoder is an amazing technique that generalizes PCA.</p>
</section>
</section>
</section>
<section id="discussion" class="level2">
<h2 class="anchored" data-anchor-id="discussion">Discussion</h2>
<p>In my opinion, what happened is that a lot of old, well-established stuff had its name changed and was presented as new, and this ended up overshadowing the truly relevant contributions of research around neural networks and deep learning.</p>
<p>You can see that Deep Learning has generalized quite a lot, so I declare this post a success if what’s written above sparked curiosity to learn more about deep learning and add it to the work that was already being done. It’s more worthwhile to bring all statistics and machine learning practitioners together in this novelty than to divide ourselves.</p>
<p>I believe that, more than ever, theoretical foundations and interpretation will have their value boosted by the spread of deep learning. With the myth that deep learning is a panacea, and with how easy it has made building a predictive model, there’s a risk of being buried by black boxes made by people who neglect important aspects like interpretability, causality and generalization. Maybe bayesianism will emerge (once again) as the solution to qualitative problems in an increasingly obscure world, shedding light on the excesses of complex models and the benefits of simple ones.</p>
<p>Speaking of bayesianism (and inference in general), the results already obtained for linear models still apply to deep learning. And we also have the advantage that all the other tools used in deep learning that don’t affect the linearity of the parameters can be used, like convolutional, recurrent, max pooling, drop out, autoencoder and so many others.</p>
<p>To wrap up, in practice I suggest applying deep learning with Keras, an amazing package that uses tensorflow or theano behind the scenes. I believe you’ll see many posts about it here! (feel free to pester Dan Falbel, one of the partners of <a href="http://curso-r.com">curso-r.com</a>, who is involved in developing this R package =]).</p>
</section>
<section id="fun-facts" class="level2">
<h2 class="anchored" data-anchor-id="fun-facts">Fun facts</h2>
<section id="n-th-smallest-deep-learning" class="level3">
<h3 class="anchored" data-anchor-id="n-th-smallest-deep-learning">N-th smallest deep learning</h3>
<p>Above we saw the smallest and the second smallest Deep Learnings (which have nothing deep about them =P). But we can go as deep as we want! The neural network representation comes out easily:</p>
<div class="cell">
<div class="cell-output-display">
<div>
<figure class="figure">
<p><img src="https://athospd.github.io/athosdamiani/posts/2017-07-29-segundo-menor-dl/dl_inf.webp" class="img-fluid figure-img"></p>
</figure>
</div>
</div>
</div>
<p>The mathematical representation, on the other hand, gets weird:</p>
<p><img src="https://latex.codecogs.com/png.latex?E%5BY%7Cx%5D%20=%20%5Cfrac%7B1%7D%7B1%20+%20%5Cexp%7B%5Cleft(%5Cbeta_%7Bp-1%7D%20+%20%5Cbeta_p%5Cfrac%7B1%7D%7B%5Cfrac%7B%5Cvdots%7D%7B1%20+%20%5Cexp%7B%5Cleft(%5Cbeta_%7B6%7D%20+%20%5Cbeta_%7B7%7D%5Cfrac%7B1%7D%7B1%20+%20%5Cexp%7B%5Cleft(%5Cbeta_4%20+%20%5Cbeta_5%5Cfrac%7B1%7D%7B1%20+%20%5Cexp%7B%5Cleft(%5Cbeta_2%20+%20%5Cbeta_3%5Cfrac%7B1%7D%7B1%20+%20%5Cexp%7B%5Cleft(%5Cbeta_0%20+%20%5Cbeta_1x%5Cright)%7D%7D%5Cright)%7D%7D%5Cright)%7D%7D%5Cright)%7D%7D%7D%5Cright)%7D%7D"></p>
</section>
<section id="vocabulary" class="level3">
<h3 class="anchored" data-anchor-id="vocabulary">Vocabulary</h3>
<p>Deep learning jargon and terms were inherited from a context different from the predictive modeling studied in statistics, and so countless synonyms came up. Some of them are:</p>
<ul>
<li>activation function = link function</li>
<li>Softmax = multinomial likelihood</li>
<li>sigmoid = S-shaped function (in tensorflow the default is the logistic)</li>
<li>weights = parameters/betas/coefficients</li>
<li>binary crossentropy = deviance of the binomial distribution (logistic regression)</li>
</ul>
<p>That’s it, we have to stay curious, ask questions and talk to each other. Cheers!</p>


</section>
</section>

 ]]></description>
  <category>concepts</category>
  <category>deep learning</category>
  <category>neural networks</category>
  <category>logistic regression</category>
  <category>keras</category>
  <category>glm</category>
  <guid>https://athospd.github.io/athosdamiani/posts/2017-07-29-segundo-menor-dl/</guid>
  <pubDate>Sat, 29 Jul 2017 00:00:00 GMT</pubDate>
  <media:content url="https://athospd.github.io/athosdamiani/posts/2017-07-29-segundo-menor-dl/what_if_i_told_you.webp" medium="image" type="image/webp"/>
</item>
<item>
  <title>WoE in R with tidywoe</title>
  <dc:creator>Athos Damiani</dc:creator>
  <link>https://athospd.github.io/athosdamiani/posts/2017-07-06-tidywoe/</link>
  <description><![CDATA[ 





<p>WoE (weight of evidence) is a tool widely used in logistic regression applications, mainly in credit scoring. Simply put, it turns categories into numbers that reflect the difference between them by how well they separate <strong>Y = 1</strong> from <strong>Y = 0</strong>.</p>
<p>If you don’t know what it is yet or want to read more about it, here is a text I enjoyed reading:</p>
<ul>
<li><a href="http://multithreaded.stitchfix.com/blog/2015/08/13/weight-of-evidence/">Data Exploration with Weight of Evidence and Information Value in R</a></li>
</ul>
<p>The author of this text is Kim Larsen, creator of the <a href="https://cran.r-project.org/web/packages/Information/index.html">Information</a> package, which is complete and full of sophisticated tools around WoE. However, in my day-to-day work every now and then I had to write my own routines to build the WoE versions of my variables, even with several complete packages available. The main reason was that they weren’t very practical and didn’t fit the tidyverse philosophy. So I ended up gathering these routines into a package called <code>tidywoe</code> and putting it out there. The idea is to help the analyst gain time, readability and reproducibility. Here is how to use it.</p>
<section id="installation" class="level2">
<h2 class="anchored" data-anchor-id="installation">Installation</h2>
<p>To install it, just run the code below.</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb1" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb1-1"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># install.packages("devtools")</span></span>
<span id="cb1-2">devtools<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">::</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">install_github</span>(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"athospd/tidywoe"</span>)</span></code></pre></div></div>
</div>
</section>
<section id="data-and-packages-for-the-examples" class="level2">
<h2 class="anchored" data-anchor-id="data-and-packages-for-the-examples">Data and packages for the examples</h2>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb2" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb2-1"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">library</span>(forcats)</span>
<span id="cb2-2"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">library</span>(ggplot2)</span>
<span id="cb2-3"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">library</span>(dplyr)</span>
<span id="cb2-4"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">library</span>(tidywoe)</span>
<span id="cb2-5"></span>
<span id="cb2-6"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># install.packages("FactoMineR")</span></span>
<span id="cb2-7"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">data</span>(tea, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">package =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"FactoMineR"</span>)</span>
<span id="cb2-8">tea_mini <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> tea <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">select</span>(breakfast, how, where, price)</span></code></pre></div></div>
</div>
</section>
<section id="how-to-use" class="level2">
<h2 class="anchored" data-anchor-id="how-to-use">How to use</h2>
<p>There are two functions that matter: - <code>add_woe()</code> - adds the woe’s to a data frame. - <code>woe_dictionary()</code> - creates a dictionary that maps the categories to their woe’s.</p>
<section id="add_woe" class="level3">
<h3 class="anchored" data-anchor-id="add_woe">add_woe()</h3>
<p>The <code>add_woe()</code> function adds the WoE versions of the variables to your data sample.</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb3" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb3-1">tea_mini <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">add_woe</span>(breakfast)</span></code></pre></div></div>
</div>
<div class="cell">
<div class="cell-output-display">
<table class="caption-top table table-sm table-striped small">
<colgroup>
<col style="width: 17%">
<col style="width: 10%">
<col style="width: 15%">
<col style="width: 14%">
<col style="width: 14%">
<col style="width: 14%">
<col style="width: 14%">
</colgroup>
<thead>
<tr class="header">
<th style="text-align: left;">breakfast</th>
<th style="text-align: left;">how</th>
<th style="text-align: left;">where</th>
<th style="text-align: left;">price</th>
<th style="text-align: right;">how_woe</th>
<th style="text-align: right;">where_woe</th>
<th style="text-align: right;">price_woe</th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td style="text-align: left;">breakfast</td>
<td style="text-align: left;">tea bag</td>
<td style="text-align: left;">chain store</td>
<td style="text-align: left;">p_unknown</td>
<td style="text-align: right;">-0.0377403</td>
<td style="text-align: right;">-0.0451204</td>
<td style="text-align: right;">-0.2564295</td>
</tr>
<tr class="even">
<td style="text-align: left;">breakfast</td>
<td style="text-align: left;">tea bag</td>
<td style="text-align: left;">chain store</td>
<td style="text-align: left;">p_variable</td>
<td style="text-align: right;">-0.0377403</td>
<td style="text-align: right;">-0.0451204</td>
<td style="text-align: right;">0.1872882</td>
</tr>
<tr class="odd">
<td style="text-align: left;">Not.breakfast</td>
<td style="text-align: left;">tea bag</td>
<td style="text-align: left;">chain store</td>
<td style="text-align: left;">p_variable</td>
<td style="text-align: right;">-0.0377403</td>
<td style="text-align: right;">-0.0451204</td>
<td style="text-align: right;">0.1872882</td>
</tr>
<tr class="even">
<td style="text-align: left;">Not.breakfast</td>
<td style="text-align: left;">tea bag</td>
<td style="text-align: left;">chain store</td>
<td style="text-align: left;">p_variable</td>
<td style="text-align: right;">-0.0377403</td>
<td style="text-align: right;">-0.0451204</td>
<td style="text-align: right;">0.1872882</td>
</tr>
</tbody>
</table>
</div>
</div>
<p>You can pick the variables you want by selecting them just like in <code>dplyr::select()</code>.</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb4" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb4-1">tea_mini <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">add_woe</span>(breakfast, where<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">:</span>price)</span></code></pre></div></div>
</div>
</section>
<section id="woe_dictionary" class="level3">
<h3 class="anchored" data-anchor-id="woe_dictionary">woe_dictionary()</h3>
<p>The <code>woe_dictionary()</code> function is one of the two parts needed to make <code>add_woe()</code> work (the other part is the data). It builds the dictionary of categories and their respective woe’s.</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb5" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb5-1">tea_mini <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">woe_dictionary</span>(breakfast)</span></code></pre></div></div>
</div>
<div class="cell">
<div class="cell-output-display">
<table class="caption-top table table-sm table-striped small">
<colgroup>
<col style="width: 8%">
<col style="width: 18%">
<col style="width: 5%">
<col style="width: 11%">
<col style="width: 15%">
<col style="width: 11%">
<col style="width: 15%">
<col style="width: 10%">
</colgroup>
<thead>
<tr class="header">
<th style="text-align: left;">variable</th>
<th style="text-align: left;">explanatory</th>
<th style="text-align: right;">n_tot</th>
<th style="text-align: right;">n_breakfast</th>
<th style="text-align: right;">n_Not.breakfast</th>
<th style="text-align: right;">p_breakfast</th>
<th style="text-align: right;">p_Not.breakfast</th>
<th style="text-align: right;">woe</th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td style="text-align: left;">how</td>
<td style="text-align: left;">tea bag</td>
<td style="text-align: right;">170</td>
<td style="text-align: right;">80</td>
<td style="text-align: right;">90</td>
<td style="text-align: right;">0.5555556</td>
<td style="text-align: right;">0.5769231</td>
<td style="text-align: right;">-0.0377403</td>
</tr>
<tr class="even">
<td style="text-align: left;">how</td>
<td style="text-align: left;">tea bag+unpackaged</td>
<td style="text-align: right;">94</td>
<td style="text-align: right;">50</td>
<td style="text-align: right;">44</td>
<td style="text-align: right;">0.3472222</td>
<td style="text-align: right;">0.2820513</td>
<td style="text-align: right;">0.2078761</td>
</tr>
<tr class="odd">
<td style="text-align: left;">how</td>
<td style="text-align: left;">unpackaged</td>
<td style="text-align: right;">36</td>
<td style="text-align: right;">14</td>
<td style="text-align: right;">22</td>
<td style="text-align: right;">0.0972222</td>
<td style="text-align: right;">0.1410256</td>
<td style="text-align: right;">-0.3719424</td>
</tr>
<tr class="even">
<td style="text-align: left;">where</td>
<td style="text-align: left;">chain store</td>
<td style="text-align: right;">192</td>
<td style="text-align: right;">90</td>
<td style="text-align: right;">102</td>
<td style="text-align: right;">0.6250000</td>
<td style="text-align: right;">0.6538462</td>
<td style="text-align: right;">-0.0451204</td>
</tr>
</tbody>
</table>
</div>
</div>
</section>
<section id="using-a-custom-dictionary" class="level3">
<h3 class="anchored" data-anchor-id="using-a-custom-dictionary">Using a custom dictionary</h3>
<p>Often we want to adjust some woe values by hand to fix the order of the effects of a given ordinal variable. That’s why <code>add_woe()</code> can take a dictionary passed by the user. This is done through the <code>.woe_dictionary</code> argument.</p>
<p>The easiest way to do this is to build an initial dictionary with <code>woe_dictionary()</code> and then change its values to get the adjustments you want. Example:</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb6" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb6-1"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Build an initial dictionary</span></span>
<span id="cb6-2">tea_mini_woe_dic <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> tea_mini <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">woe_dictionary</span>(breakfast)</span>
<span id="cb6-3"></span>
<span id="cb6-4"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Tweak the woes a little</span></span>
<span id="cb6-5">tea_mini_woe_dic_arrumado <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> tea_mini_woe_dic <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">mutate</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">woe =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">if_else</span>(explanatory <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">==</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"p_unknown"</span>, <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">0</span>, woe))</span>
<span id="cb6-6"></span>
<span id="cb6-7"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Pass this dictionary to add_woe()</span></span>
<span id="cb6-8">tea_mini <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">add_woe</span>(breakfast, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">.woe_dictionary =</span> tea_mini_woe_dic_arrumado)</span></code></pre></div></div>
</div>
<div class="cell">
<div class="cell-output-display">
<table class="caption-top table table-sm table-striped small">
<colgroup>
<col style="width: 18%">
<col style="width: 10%">
<col style="width: 15%">
<col style="width: 14%">
<col style="width: 14%">
<col style="width: 14%">
<col style="width: 12%">
</colgroup>
<thead>
<tr class="header">
<th style="text-align: left;">breakfast</th>
<th style="text-align: left;">how</th>
<th style="text-align: left;">where</th>
<th style="text-align: left;">price</th>
<th style="text-align: right;">how_woe</th>
<th style="text-align: right;">where_woe</th>
<th style="text-align: right;">price_woe</th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td style="text-align: left;">breakfast</td>
<td style="text-align: left;">tea bag</td>
<td style="text-align: left;">chain store</td>
<td style="text-align: left;">p_unknown</td>
<td style="text-align: right;">-0.0377403</td>
<td style="text-align: right;">-0.0451204</td>
<td style="text-align: right;">0.0000000</td>
</tr>
<tr class="even">
<td style="text-align: left;">breakfast</td>
<td style="text-align: left;">tea bag</td>
<td style="text-align: left;">chain store</td>
<td style="text-align: left;">p_variable</td>
<td style="text-align: right;">-0.0377403</td>
<td style="text-align: right;">-0.0451204</td>
<td style="text-align: right;">0.1872882</td>
</tr>
<tr class="odd">
<td style="text-align: left;">Not.breakfast</td>
<td style="text-align: left;">tea bag</td>
<td style="text-align: left;">chain store</td>
<td style="text-align: left;">p_variable</td>
<td style="text-align: right;">-0.0377403</td>
<td style="text-align: right;">-0.0451204</td>
<td style="text-align: right;">0.1872882</td>
</tr>
<tr class="even">
<td style="text-align: left;">Not.breakfast</td>
<td style="text-align: left;">tea bag</td>
<td style="text-align: left;">chain store</td>
<td style="text-align: left;">p_variable</td>
<td style="text-align: right;">-0.0377403</td>
<td style="text-align: right;">-0.0451204</td>
<td style="text-align: right;">0.1872882</td>
</tr>
</tbody>
</table>
</div>
</div>
</section>
</section>
<section id="exploration-example" class="level2">
<h2 class="anchored" data-anchor-id="exploration-example">Exploration example</h2>
<p><code>woe_dictionary()</code> returns a tidy table, very convenient for further exploration. For example, the table is ready for ggplot.</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb7" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb7-1">tea_mini_woe_dic_arrumado <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb7-2">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">mutate</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">explanatory =</span> explanatory <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span> as.factor <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">fct_reorder</span>(woe)) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb7-3">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">filter</span>(variable <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%in%</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">c</span>(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"price"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"how"</span>)) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb7-4">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">ggplot</span>() <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span></span>
<span id="cb7-5">    <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">geom_bar</span>(<span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">aes</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">x =</span> explanatory, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">y =</span> woe), <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">stat =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"identity"</span>) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span></span>
<span id="cb7-6">    <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">facet_wrap</span>(<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">~</span>variable, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">scales =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"free_x"</span>) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span></span>
<span id="cb7-7">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">theme</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">axis.text.x =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">element_text</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">angle =</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">30</span>))</span></code></pre></div></div>
<div class="cell-output-display">
<div>
<figure class="figure">
<p><img src="https://athospd.github.io/athosdamiani/posts/2017-07-06-tidywoe/index_files/figure-html/unnamed-chunk-12-1.png" class="img-fluid figure-img" width="672"></p>
</figure>
</div>
</div>
</div>
<p>Here is the <a href="https://github.com/Athospd/tidywoe">package’s github</a> for contributions. I still plan to add a lot of new stuff to the package.</p>
<p>Cheers</p>


</section>

 ]]></description>
  <category>tutorials</category>
  <category>woe</category>
  <guid>https://athospd.github.io/athosdamiani/posts/2017-07-06-tidywoe/</guid>
  <pubDate>Thu, 06 Jul 2017 00:00:00 GMT</pubDate>
  <media:content url="https://athospd.github.io/athosdamiani/posts/2017-07-06-tidywoe/tidywoe_hex2.webp" medium="image" type="image/webp"/>
</item>
<item>
  <title>Principal Components: Intuition</title>
  <dc:creator>Athos Damiani</dc:creator>
  <link>https://athospd.github.io/athosdamiani/posts/2017-06-24-componentes-principais-intuicao/</link>
  <description><![CDATA[ 





<p>Principal components are widely used in statistical modeling, but their rigorous mathematical definition makes PCA look like a more abstract concept than we can grasp, especially when people say <strong>“maximize the total variance”</strong> and <strong>“reduce the dimensionality”</strong>. The first reaction is a big “HUH?”.</p>
<p>Below are some 3D plots that give a good illustration of what these two statements mean and, even though it’s a simple example, they’ll help extrapolate the idea to more complex problems.</p>
<ol type="1">
<li>The first plot is the representation with <strong>100% of the information</strong> in three dimensions.</li>
<li>The second plot is how the same dataset looks with one dimension less (<strong>dimensionality reduction</strong>).</li>
<li>When we drop a dimension, we lose information (it’s no longer <strong>100% of the total variance</strong>), and this loss is measured by the variance explained by the dimensions we left behind.</li>
<li>The first two dimensions project the data so that they show the <strong>maximum total variance</strong> possible in only two dimensions.</li>
</ol>
<p><strong>packages for the examples</strong></p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb1" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb1-1"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">library</span>(magrittr)</span>
<span id="cb1-2"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">library</span>(tidyr)</span>
<span id="cb1-3"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">library</span>(dplyr)</span></code></pre></div></div>
</div>
<p>Note: don’t forget to rotate the plots! =)</p>
<section id="example-1---tetrahedron" class="level2">
<h2 class="anchored" data-anchor-id="example-1---tetrahedron">Example 1 - Tetrahedron</h2>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb2" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb2-1">a <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span></span>
<span id="cb2-2">tetraedro <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">data.frame</span>( <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">x =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">c</span>(a <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">*</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">sqrt</span>(<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">3</span>)<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">/</span><span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">3</span>, <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">-</span> a <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">*</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">sqrt</span>(<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">3</span>)<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">/</span><span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">6</span>, <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">-</span> a <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">*</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">sqrt</span>(<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">3</span>)<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">/</span><span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">6</span>, <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">0</span>),</span>
<span id="cb2-3">                         <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">y =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">c</span>(<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">0</span>, <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">-</span> a<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">/</span><span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">2</span>, a<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">/</span><span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">2</span>, <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">0</span>),</span>
<span id="cb2-4">                         <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">z =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">c</span>(<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">0</span>, <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">0</span>, <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">0</span>, a <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">*</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">sqrt</span>(<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">6</span>)<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">/</span><span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">3</span>),</span>
<span id="cb2-5">                         <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">cor =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">c</span>(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"a"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"b"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"c"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"d"</span>),</span>
<span id="cb2-6">                         <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">id =</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">:</span><span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">4</span>)</span>
<span id="cb2-7"></span>
<span id="cb2-8">tetraedro_linhas <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">combn</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">x =</span> tetraedro<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span>id, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">m =</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">2</span>) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb2-9">  t <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb2-10">  as.data.frame.matrix <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb2-11">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">set_names</span>(<span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">c</span>(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"id1"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"id2"</span>)) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb2-12">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">mutate</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">id_par =</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">:</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">n</span>()) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb2-13">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">gather</span>(id_ordem, id, id1, id2) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb2-14">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">left_join</span>(tetraedro, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">by =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"id"</span>) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb2-15">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">arrange</span>(id_ordem)</span></code></pre></div></div>
</div>
<p><strong>Contribution of the components to the total variance</strong></p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb3" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb3-1">tetraedro_pc <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">prcomp</span>(tetraedro <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span> dplyr<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">::</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">select</span>(x, y, z)) <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># PCA happens here</span></span>
<span id="cb3-2">knitr<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">::</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">kable</span>(<span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">summary</span>(tetraedro_pc)<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span>importance)</span></code></pre></div></div>
<div class="cell-output-display">
<table class="caption-top table table-sm table-striped small">
<thead>
<tr class="header">
<th style="text-align: left;"></th>
<th style="text-align: right;">PC1</th>
<th style="text-align: right;">PC2</th>
<th style="text-align: right;">PC3</th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td style="text-align: left;">Standard deviation</td>
<td style="text-align: right;">0.4082483</td>
<td style="text-align: right;">0.4082483</td>
<td style="text-align: right;">0.4082483</td>
</tr>
<tr class="even">
<td style="text-align: left;">Proportion of Variance</td>
<td style="text-align: right;">0.3333300</td>
<td style="text-align: right;">0.3333300</td>
<td style="text-align: right;">0.3333300</td>
</tr>
<tr class="odd">
<td style="text-align: left;">Cumulative Proportion</td>
<td style="text-align: right;">0.3333300</td>
<td style="text-align: right;">0.6666700</td>
<td style="text-align: right;">1.0000000</td>
</tr>
</tbody>
</table>
</div>
</div>
<p><strong>3 dimensions (100% of the variance)</strong></p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb4" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb4-1"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">library</span>(plotly)</span>
<span id="cb4-2"></span>
<span id="cb4-3"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">plot_ly</span>(tetraedro_linhas, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">x =</span> <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">~</span>x, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">y =</span> <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">~</span>y, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">z =</span> <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">~</span>z) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb4-4">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">add_lines</span>() <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb4-5">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">add_markers</span>()</span></code></pre></div></div>
<div class="cell-output-display">
<div class="plotly html-widget html-fill-item" id="htmlwidget-f723208b5186d5c083a4" style="width:100%;height:464px;"></div>
<script type="application/json" data-for="htmlwidget-f723208b5186d5c083a4">{"x":{"visdat":{"1d554a51a46":["function () ","plotlyVisDat"]},"cur_data":"1d554a51a46","attrs":{"1d554a51a46":{"x":{},"y":{},"z":{},"alpha_stroke":1,"sizes":[10,100],"spans":[1,20],"type":"scatter3d","mode":"lines","inherit":true},"1d554a51a46.1":{"x":{},"y":{},"z":{},"alpha_stroke":1,"sizes":[10,100],"spans":[1,20],"type":"scatter3d","mode":"markers","inherit":true}},"layout":{"margin":{"b":40,"l":60,"t":25,"r":10},"scene":{"xaxis":{"title":"x"},"yaxis":{"title":"y"},"zaxis":{"title":"z"}},"hovermode":"closest","showlegend":true},"source":"A","config":{"modeBarButtonsToAdd":["hoverclosest","hovercompare"],"showSendToCloud":false},"data":[{"x":[-0.28867513459481287,-0.28867513459481287,-0.28867513459481287,-0.28867513459481287,-0.28867513459481287,-0.28867513459481287,0,0,0,0.57735026918962573,0.57735026918962573,0.57735026918962573],"y":[-0.5,-0.5,0.5,-0.5,0.5,0.5,0,0,0,0,0,0],"z":[0,0,0,0,0,0,0.81649658092772592,0.81649658092772592,0.81649658092772592,0,0,0],"type":"scatter3d","mode":"lines","marker":{"color":"rgba(31,119,180,1)","line":{"color":"rgba(31,119,180,1)"}},"error_y":{"color":"rgba(31,119,180,1)"},"error_x":{"color":"rgba(31,119,180,1)"},"line":{"color":"rgba(31,119,180,1)"},"frame":null},{"x":[0.57735026918962573,0.57735026918962573,0.57735026918962573,-0.28867513459481287,-0.28867513459481287,-0.28867513459481287,-0.28867513459481287,-0.28867513459481287,0,-0.28867513459481287,0,0],"y":[0,0,0,-0.5,-0.5,0.5,-0.5,0.5,0,0.5,0,0],"z":[0,0,0,0,0,0,0,0,0.81649658092772592,0,0.81649658092772592,0.81649658092772592],"type":"scatter3d","mode":"markers","marker":{"color":"rgba(255,127,14,1)","line":{"color":"rgba(255,127,14,1)"}},"error_y":{"color":"rgba(255,127,14,1)"},"error_x":{"color":"rgba(255,127,14,1)"},"line":{"color":"rgba(255,127,14,1)"},"frame":null}],"highlight":{"on":"plotly_click","persistent":false,"dynamic":false,"selectize":false,"opacityDim":0.20000000000000001,"selected":{"opacity":1},"debounce":0},"shinyEvents":["plotly_hover","plotly_click","plotly_selected","plotly_relayout","plotly_brushed","plotly_brushing","plotly_clickannotation","plotly_doubleclick","plotly_deselect","plotly_afterplot","plotly_sunburstclick"],"base_url":"https://plot.ly"},"evals":[],"jsHooks":[]}</script>
</div>
</div>
<p><strong>2 dimensions (67% of the variance)</strong></p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb5" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb5-1">tetraedro_pc_pred <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> tetraedro_pc <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span> predict <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span> as.data.frame</span>
<span id="cb5-2"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">plot_ly</span>(tetraedro_pc_pred, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">x =</span> <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">~</span>PC2, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">y =</span> <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">~</span>PC3, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">z =</span> <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">~</span>PC3) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb5-3">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">add_lines</span>() <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb5-4">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">add_markers</span>()</span></code></pre></div></div>
<div class="cell-output-display">
<div class="plotly html-widget html-fill-item" id="htmlwidget-54a7f0f2a9cff9827191" style="width:100%;height:464px;"></div>
<script type="application/json" data-for="htmlwidget-54a7f0f2a9cff9827191">{"x":{"visdat":{"1d55414c353b2":["function () ","plotlyVisDat"]},"cur_data":"1d55414c353b2","attrs":{"1d55414c353b2":{"x":{},"y":{},"z":{},"alpha_stroke":1,"sizes":[10,100],"spans":[1,20],"type":"scatter3d","mode":"lines","inherit":true},"1d55414c353b2.1":{"x":{},"y":{},"z":{},"alpha_stroke":1,"sizes":[10,100],"spans":[1,20],"type":"scatter3d","mode":"markers","inherit":true}},"layout":{"margin":{"b":40,"l":60,"t":25,"r":10},"scene":{"xaxis":{"title":"PC2"},"yaxis":{"title":"PC3"},"zaxis":{"title":"PC3"}},"hovermode":"closest","showlegend":true},"source":"A","config":{"modeBarButtonsToAdd":["hoverclosest","hovercompare"],"showSendToCloud":false},"data":[{"x":[-0.20412414523193148,-0.20412414523193148,-0.20412414523193148,0.61237243569579447],"y":[-0.57735026918962573,0.28867513459481287,0.28867513459481287,0],"z":[-0.57735026918962573,0.28867513459481287,0.28867513459481287,0],"type":"scatter3d","mode":"lines","marker":{"color":"rgba(31,119,180,1)","line":{"color":"rgba(31,119,180,1)"}},"error_y":{"color":"rgba(31,119,180,1)"},"error_x":{"color":"rgba(31,119,180,1)"},"line":{"color":"rgba(31,119,180,1)"},"frame":null},{"x":[-0.20412414523193148,-0.20412414523193148,-0.20412414523193148,0.61237243569579447],"y":[-0.57735026918962573,0.28867513459481287,0.28867513459481287,0],"z":[-0.57735026918962573,0.28867513459481287,0.28867513459481287,0],"type":"scatter3d","mode":"markers","marker":{"color":"rgba(255,127,14,1)","line":{"color":"rgba(255,127,14,1)"}},"error_y":{"color":"rgba(255,127,14,1)"},"error_x":{"color":"rgba(255,127,14,1)"},"line":{"color":"rgba(255,127,14,1)"},"frame":null}],"highlight":{"on":"plotly_click","persistent":false,"dynamic":false,"selectize":false,"opacityDim":0.20000000000000001,"selected":{"opacity":1},"debounce":0},"shinyEvents":["plotly_hover","plotly_click","plotly_selected","plotly_relayout","plotly_brushed","plotly_brushing","plotly_clickannotation","plotly_doubleclick","plotly_deselect","plotly_afterplot","plotly_sunburstclick"],"base_url":"https://plot.ly"},"evals":[],"jsHooks":[]}</script>
</div>
</div>
</section>
<section id="example-2---a-cool-shape" class="level2">
<h2 class="anchored" data-anchor-id="example-2---a-cool-shape">Example 2 - A cool shape</h2>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb6" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb6-1">x <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">c</span>()</span>
<span id="cb6-2">y <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">c</span>()</span>
<span id="cb6-3">z <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">c</span>()</span>
<span id="cb6-4">c <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">c</span>()</span>
<span id="cb6-5"></span>
<span id="cb6-6"><span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">for</span> (i <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">in</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">:</span><span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">62</span>) {</span>
<span id="cb6-7">  r <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">20</span> <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">*</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">cos</span>(i <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">/</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">20</span>)</span>
<span id="cb6-8">  x <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">c</span>(x, r <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">*</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">cos</span>(i))</span>
<span id="cb6-9">  y <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">c</span>(y, r <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">*</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">sin</span>(i))</span>
<span id="cb6-10">  z <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">c</span>(z, i)</span>
<span id="cb6-11">  c <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">c</span>(c, i)</span>
<span id="cb6-12">}</span>
<span id="cb6-13"></span>
<span id="cb6-14">forma_legal <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">data.frame</span>(x, y, z, c)</span></code></pre></div></div>
</div>
<p><strong>Contribution of the components to the total variance</strong></p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb7" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb7-1">forma_legal_pc <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">prcomp</span>(forma_legal <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span> dplyr<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">::</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">select</span>(x, y, z)) <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># PCA happens here</span></span>
<span id="cb7-2">knitr<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">::</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">kable</span>(<span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">summary</span>(forma_legal_pc)<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span>importance)</span></code></pre></div></div>
<div class="cell-output-display">
<table class="caption-top table table-sm table-striped small">
<thead>
<tr class="header">
<th style="text-align: left;"></th>
<th style="text-align: right;">PC1</th>
<th style="text-align: right;">PC2</th>
<th style="text-align: right;">PC3</th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td style="text-align: left;">Standard deviation</td>
<td style="text-align: right;">18.04936</td>
<td style="text-align: right;">10.13750</td>
<td style="text-align: right;">9.854732</td>
</tr>
<tr class="even">
<td style="text-align: left;">Proportion of Variance</td>
<td style="text-align: right;">0.61975</td>
<td style="text-align: right;">0.19550</td>
<td style="text-align: right;">0.184750</td>
</tr>
<tr class="odd">
<td style="text-align: left;">Cumulative Proportion</td>
<td style="text-align: right;">0.61975</td>
<td style="text-align: right;">0.81525</td>
<td style="text-align: right;">1.000000</td>
</tr>
</tbody>
</table>
</div>
</div>
<p><strong>3 dimensions (100% of the variance)</strong></p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb8" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb8-1"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">plot_ly</span>(forma_legal, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">x =</span> <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">~</span>x, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">y =</span> <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">~</span>y, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">z =</span> <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">~</span>z, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">type =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"scatter3d"</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">mode =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"markers+lines"</span>,</span>
<span id="cb8-2">        <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">line =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">list</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">width =</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">6</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">color =</span> <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">~</span>c, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">colorscale =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">'Viridis'</span>),</span>
<span id="cb8-3">        <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">marker =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">list</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">size =</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">3.5</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">color =</span> <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">~</span>c, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">colorscale =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">'Greens'</span>,</span>
<span id="cb8-4">                      <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">cmin =</span> <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">-</span><span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">20</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">cmax =</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">50</span>))</span></code></pre></div></div>
<div class="cell-output-display">
<div class="plotly html-widget html-fill-item" id="htmlwidget-cfee95381032b5f7f88d" style="width:100%;height:464px;"></div>
<script type="application/json" data-for="htmlwidget-cfee95381032b5f7f88d">{"x":{"visdat":{"1d554ebb23bd":["function () ","plotlyVisDat"]},"cur_data":"1d554ebb23bd","attrs":{"1d554ebb23bd":{"x":{},"y":{},"z":{},"mode":"markers+lines","line":{"width":6,"color":{},"colorscale":"Viridis"},"marker":{"size":3.5,"color":{},"colorscale":"Greens","cmin":-20,"cmax":50},"alpha_stroke":1,"sizes":[10,100],"spans":[1,20],"type":"scatter3d"}},"layout":{"margin":{"b":40,"l":60,"t":25,"r":10},"scene":{"xaxis":{"title":"x"},"yaxis":{"title":"y"},"zaxis":{"title":"z"}},"hovermode":"closest","showlegend":false},"source":"A","config":{"modeBarButtonsToAdd":["hoverclosest","hovercompare"],"showSendToCloud":false},"data":[{"x":[10.792541373556107,-8.2813567146336098,-19.577518960244319,-12.812285332551163,5.4968763012981725,18.345714212225747,14.163904117684492,-2.6802881153442994,-16.40849208359786,-14.72709084192371,0.075460321240854236,13.929254518488953,14.448073615159741,2.0916478611455167,-11.117103793135696,-13.344155704224436,-3.6320633095901154,8.2091889612038447,11.502255137480622,4.4097535796240086,-5.4506844394149923,-9.07156704860245,-4.353111937891315,3.0740910518955737,6.2509682449924702,3.4610032187504634,-1.2796070534421031,-3.2722273767265659,-1.8028600794727121,0.21822631836467205,0.38043819629479975,-0.48717847238731893,0.021009360815016032,2.1866721585767368,3.2215914215750581,0.58147236656004808,-4.2188129499713556,-6.1753068941359555,-1.9741220474989305,5.5508832905894359,9.1047035597969952,4.0386205633803227,-6.0769104159475082,-11.768178084388957,-6.5998683362309505,5.7589960320544238,13.946542214371545,9.4407682592777888,-4.6305154384986338,-15.461527458922742,-12.32055429509874,2.7932993393332683,16.190835102122207,14.995118339847984,-0.40903626888692884,-16.078460940078756,-17.237619247369775,-2.3143785127533052,15.139423071678529,18.857634085517432,5.1403951528895373,-13.458493596766125],"y":[16.808387303833907,18.095094543362887,2.7907076497539571,-14.834336633563069,-18.5822728240197,-5.3387164211328777,12.343105670856822,18.225185801197672,7.4218179172382825,-9.5484688043347941,-17.050323458076498,-8.8570547844353875,6.6897634140410664,15.153165933395359,9.516167484139217,-4.0116834472770515,-12.690122822732031,-9.3364231727483027,1.7436207671394468,9.8653284819900851,8.3259124558181465,-0.080298391273307554,-6.9134081465094557,-6.5628668351876893,-0.83466933015956046,4.0796698452214821,4.1890544739080404,0.92090165667652057,-1.5993944178736981,-1.3978118449504959,-0.16803786520920139,-0.32202791352884846,-1.5822783020016302,-1.3633878226730454,1.5264374387031803,4.5066846594916825,3.5470566602568367,-1.9162573888581858,-7.1355715469435657,-6.2015296920624854,1.4627316163828821,9.2540466649407875,9.1055656266128544,-0.20835205404212692,-10.690302991882886,-12.016799046159107,-1.736728985816248,11.330123183906796,14.692202035717482,4.2039987784992654,-11.126521936531649,-16.908601747659556,-6.9808112660148298,10.103712246619141,18.481521695709663,9.8283401192882849,-8.355060720936681,-19.280756092227548,-12.50174208682674,6.0352045555063176,19.241362357189434,14.770828328514572],"z":[1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62],"mode":"markers+lines","line":{"color":[1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62],"width":6,"colorscale":"Viridis"},"marker":{"color":[1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62],"size":3.5,"colorscale":"Greens","cmin":-20,"cmax":50,"line":{"color":"rgba(31,119,180,1)"}},"type":"scatter3d","error_y":{"color":"rgba(31,119,180,1)"},"error_x":{"color":"rgba(31,119,180,1)"},"frame":null}],"highlight":{"on":"plotly_click","persistent":false,"dynamic":false,"selectize":false,"opacityDim":0.20000000000000001,"selected":{"opacity":1},"debounce":0},"shinyEvents":["plotly_hover","plotly_click","plotly_selected","plotly_relayout","plotly_brushed","plotly_brushing","plotly_clickannotation","plotly_doubleclick","plotly_deselect","plotly_afterplot","plotly_sunburstclick"],"base_url":"https://plot.ly"},"evals":[],"jsHooks":[]}</script>
</div>
</div>
<p><strong>2 dimensions (82% of the variance)</strong></p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb9" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb9-1">forma_legal_pc_pred <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> forma_legal_pc <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span> predict <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span> as.data.frame</span>
<span id="cb9-2"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">plot_ly</span>(forma_legal_pc_pred, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">x =</span> <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">~</span>PC1, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">y =</span> <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">~</span>PC2, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">z =</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">type =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">'scatter3d'</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">mode =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">'lines+markers'</span>,</span>
<span id="cb9-3">        <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">line =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">list</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">width =</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">6</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">color =</span> <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">~</span>c, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">colorscale =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">'Viridis'</span>),</span>
<span id="cb9-4">        <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">marker =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">list</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">size =</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">3.5</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">color =</span> <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">~</span>c, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">colorscale =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">'Greens'</span>,</span>
<span id="cb9-5">                      <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">cmin =</span> <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">-</span><span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">20</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">cmax =</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">50</span>))</span></code></pre></div></div>
<div class="cell-output-display">
<div class="plotly html-widget html-fill-item" id="htmlwidget-544d732a4e2545e623ac" style="width:100%;height:464px;"></div>
<script type="application/json" data-for="htmlwidget-544d732a4e2545e623ac">{"x":{"visdat":{"1d55419e5145d":["function () ","plotlyVisDat"]},"cur_data":"1d55419e5145d","attrs":{"1d55419e5145d":{"x":{},"y":{},"z":1,"mode":"lines+markers","line":{"width":6,"color":{},"colorscale":"Viridis"},"marker":{"size":3.5,"color":{},"colorscale":"Greens","cmin":-20,"cmax":50},"alpha_stroke":1,"sizes":[10,100],"spans":[1,20],"type":"scatter3d"}},"layout":{"margin":{"b":40,"l":60,"t":25,"r":10},"scene":{"xaxis":{"title":"PC1"},"yaxis":{"title":"PC2"},"zaxis":{"title":[]}},"hovermode":"closest","showlegend":false},"source":"A","config":{"modeBarButtonsToAdd":["hoverclosest","hovercompare"],"showSendToCloud":false},"data":[{"x":[-30.056063248044843,-29.717298190478211,-29.156336835555514,-27.973056882292369,-26.346910771292041,-24.861062420788365,-23.955409358152085,-23.525480662871946,-23.036015821015276,-22.02789847573014,-20.534923268341075,-19.028901333361159,-17.965781133096179,-17.371948631061183,-16.849216601992417,-15.967132675746765,-14.65492934540841,-13.233223736127732,-12.08658638185684,-11.310398687705529,-10.659000418535417,-9.8104262176636645,-8.666747142002972,-7.4076164281344585,-6.2807440813417834,-5.3641054620409934,-4.5295254572558861,-3.6091439460610784,-2.565974356809849,-1.4955928304117123,-0.48694195026766607,0.48177286245790901,1.4951622231385175,2.5706306228081037,3.6145559812003265,4.5307339814975611,5.3600957474455138,6.2752887036841116,7.4056892018605724,8.6699403503148158,9.8156287942155664,10.661432568735018,11.308030187194978,12.081820656351873,13.23045330930281,14.656443721867413,15.971221472385007,16.852036213461954,17.371152896171289,17.962461471759312,19.026236021226687,20.535105644046407,22.030328791372856,23.03824249259717,23.525661822737693,23.953822823004373,24.859431936327255,26.346555688423717,27.973823471109185,29.157170037739348,29.71753017324993,30.055932883756292],"y":[-15.225275818464533,-18.109729668391502,-3.8083672831796389,14.322690004206434,19.595432633505311,7.4780490874895378,-10.492565139294003,-17.768843395903843,-8.1568932437882182,8.894655546020136,17.613476535953531,10.612950750339007,-4.8353316017819363,-14.306815231220073,-9.7993369169380635,3.4936158174205976,12.957240824693894,10.610115202151217,-0.15413180407871629,-8.8429345924221856,-8.1372815337752158,-0.064953946676285512,7.1403755419682522,7.4149927899486734,1.9739150000716781,-3.1574232938763127,-3.6646539468942678,-0.57544173437617518,2.059379990306951,2.0282908459950955,0.81649057881835174,0.89705291592834957,2.1955392646839398,2.1593484724818794,-0.63332397764796811,-3.8248188717037341,-3.2718284413216701,2.0078206965345999,7.5616056851551043,7.2630006958943429,-0.075632674173061964,-8.2649821492800069,-8.9667766143829137,-0.16387245510622717,10.715190758537863,13.075199861179156,3.5192579146660674,-9.8799130643881163,-14.412160271960502,-4.8708833291689917,10.669306167530983,17.70058192388338,8.9333374240101833,-8.1912439036288021,-17.833460220511657,-10.527167709794824,7.4945872413127379,19.635285408221886,14.346432570523339,-3.812645608952892,-18.124435178956723,-15.232094527393429],"z":[1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1],"mode":"lines+markers","line":{"color":[1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62],"width":6,"colorscale":"Viridis"},"marker":{"color":[1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62],"size":3.5,"colorscale":"Greens","cmin":-20,"cmax":50,"line":{"color":"rgba(31,119,180,1)"}},"type":"scatter3d","error_y":{"color":"rgba(31,119,180,1)"},"error_x":{"color":"rgba(31,119,180,1)"},"frame":null}],"highlight":{"on":"plotly_click","persistent":false,"dynamic":false,"selectize":false,"opacityDim":0.20000000000000001,"selected":{"opacity":1},"debounce":0},"shinyEvents":["plotly_hover","plotly_click","plotly_selected","plotly_relayout","plotly_brushed","plotly_brushing","plotly_clickannotation","plotly_doubleclick","plotly_deselect","plotly_afterplot","plotly_sunburstclick"],"base_url":"https://plot.ly"},"evals":[],"jsHooks":[]}</script>
</div>
</div>
</section>
<section id="learn-more" class="level2">
<h2 class="anchored" data-anchor-id="learn-more">Learn more</h2>
<p>My favorite book on this subject (and many others) is <a href="http://www-bcf.usc.edu/~gareth/ISL/ISLR%20Seventh%20Printing.pdf">An Introduction to Statistical Learning</a>.</p>
<p>cheers</p>


</section>

 ]]></description>
  <category>concepts</category>
  <category>pca</category>
  <category>principal components</category>
  <category>plotly</category>
  <guid>https://athospd.github.io/athosdamiani/posts/2017-06-24-componentes-principais-intuicao/</guid>
  <pubDate>Sat, 24 Jun 2017 00:00:00 GMT</pubDate>
  <media:content url="https://athospd.github.io/athosdamiani/posts/2017-06-24-componentes-principais-intuicao/componentes_principais.webp" medium="image" type="image/webp"/>
</item>
<item>
  <title>Lollapalooza according to Spotify</title>
  <dc:creator>Athos Damiani</dc:creator>
  <link>https://athospd.github.io/athosdamiani/posts/2017-03-27-lollapalooza-sp-2017-segundo-spotify/</link>
  <description><![CDATA[ 





<p>On the day I went to Lollapalooza I discovered <a href="https://github.com/tiagomendesdantas/Rspotify">Rspotify</a>, a wrapper for the Spotify API, and that gave me the idea of putting together info from both.</p>
<p>The fun here will involve</p>
<ul>
<li>Web Scraping - to download and structure the Lolapalooza SP 2017 schedule tables</li>
<li>Spotify API - through the Rspotify package</li>
<li>all the tidyverse packages</li>
</ul>
<p>This year’s Lollapalooza in São Paulo had 47 bands spread over four stages. The fun part is linking the Lolla schedule to the band popularity info provided by Spotify. Below I describe how I got the data, list the three hypotheses I came up with and make some plots to discuss them.</p>
<section id="dataset" class="level2">
<h2 class="anchored" data-anchor-id="dataset">Dataset</h2>
<section id="prerequisites" class="level3">
<h3 class="anchored" data-anchor-id="prerequisites">Prerequisites</h3>
<p><strong>Packages</strong></p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb1" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb1-1"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># installs Rspotify</span></span>
<span id="cb1-2"><span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">if</span>(<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">!</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">require</span>(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Rspotify"</span>))</span>
<span id="cb1-3">  devtools<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">::</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">install_github</span>(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"tiagomendesdantas/Rspotify"</span>)</span></code></pre></div></div>
</div>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb2" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb2-1"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">library</span>(Rspotify)</span>
<span id="cb2-2"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">library</span>(magrittr)</span>
<span id="cb2-3"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">library</span>(forcats)</span>
<span id="cb2-4"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">library</span>(stringi)</span>
<span id="cb2-5"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">library</span>(lubridate)</span>
<span id="cb2-6"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">library</span>(httr)</span>
<span id="cb2-7"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">library</span>(rvest)</span>
<span id="cb2-8"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">library</span>(tidyverse)</span></code></pre></div></div>
</div>
<p><code>Rspotify</code> is a new package that isn’t on CRAN yet, but it already works.</p>
<p><strong>Spotify account</strong></p>
<p>To use the Spotify API you need an account on their website, as you can imagine.</p>
<p><strong>Spotify App</strong></p>
<p>To get an access code for their API you need to create an <em>App</em> within your Spotify account; this is the most bureaucratic prerequisite of all. I learned how to do it by following the steps in the README of the <code>Rspotify</code> package itself on Github <a href="https://github.com/tiagomendesdantas/Rspotify">(see here)</a>.</p>
<p>In the end, you’ll have an <code>app_id</code>, a <code>client_id</code> and a <code>client_secret</code> in hand.</p>
</section>
<section id="extracting-the-lollapalooza-sp-2017-schedule" class="level3">
<h3 class="anchored" data-anchor-id="extracting-the-lollapalooza-sp-2017-schedule">Extracting the Lollapalooza SP 2017 schedule</h3>
<p>The goal here is to get a <code>data.frame</code> version of the tables on the website <a href="https://www.lollapaloozabr.com/lineup-horarios/">lollapaloozabr.com/lineup-horarios/</a>. It has the full schedule for both days of the event.</p>
<p>On to the code! Tip: the best way to learn what each step of the code does is to run it line by line and look at the result.</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb3" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb3-1"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># schedule from the lollapalooza 2017 website ------------------</span></span>
<span id="cb3-2"></span>
<span id="cb3-3">lolla2017_programacao <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"https://www.lollapaloozabr.com/lineup-horarios/"</span> <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb3-4">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">GET</span>() <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb3-5">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">read_html</span>() <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb3-6">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">html_table</span>() <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb3-7">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">set_names</span>(<span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">c</span>(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"sabado"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"domingo"</span>)) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb3-8">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">at_depth</span>(<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">2</span>, <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">~</span> .x <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb3-9">             <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">stri_replace_all_regex</span>(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">" {2,}"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">""</span>) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb3-10">             <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">stri_replace_all_regex</span>(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"[</span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">\\</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">n]{1}"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">","</span>) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb3-11">             <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">stri_replace_all_regex</span>(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"[,]{2,}"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"</span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">\\\n</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"</span>) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb3-12">             <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">read.csv</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">text =</span> ., <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">header =</span> <span class="cn" style="color: #8f5902;
background-color: null;
font-style: inherit;">FALSE</span>,</span>
<span id="cb3-13">                      <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">col.names =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">c</span>(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"artist"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"hora"</span>))) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb3-14">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">map</span>(<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">~</span> .x[<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">-</span><span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span>] <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb3-15">        <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">data_frame</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">palco =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">names</span>(.), <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">programacao_palco =</span> .) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb3-16">        <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">unnest</span>(programacao_palco)) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb3-17">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">data_frame</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">dia =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">names</span>(.), <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">programacao =</span> .) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb3-18">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">unnest</span>(programacao) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb3-19">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">separate</span>(hora, <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">c</span>(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"hora_ini"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"hora_fim"</span>), <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">sep =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"-"</span>) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb3-20">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">mutate</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">artist =</span> artist <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span> tolower,</span>
<span id="cb3-21">         <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">hora_ini =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">paste</span>(<span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">if_else</span>(dia <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%in%</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"sabado"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"2017-03-25"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"2017-03-26"</span>), hora_ini) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">ymd_hm</span>(),</span>
<span id="cb3-22">         <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">hora_ini =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">if_else</span>(<span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">hour</span>(hora_ini) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">12</span>, hora_ini <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">hours</span>(<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">12</span>), hora_ini),</span>
<span id="cb3-23">         <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">hora_fim =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">paste</span>(<span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">if_else</span>(dia <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%in%</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"sabado"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"2017-03-25"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"2017-03-26"</span>), hora_fim) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">ymd_hm</span>(),</span>
<span id="cb3-24">         <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">hora_fim =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">if_else</span>(<span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">hour</span>(hora_fim) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">12</span>, hora_fim <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">hours</span>(<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">12</span>), hora_fim),</span>
<span id="cb3-25">         <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">dia =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">fct_relevel</span>(dia, <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">c</span>(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"sabado"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"domingo"</span>)))</span></code></pre></div></div>
</div>
<p><strong>Result</strong></p>
<div class="cell">
<div class="cell-output-display">
<table class="caption-top table table-sm table-striped small">
<colgroup>
<col style="width: 9%">
<col style="width: 14%">
<col style="width: 23%">
<col style="width: 26%">
<col style="width: 26%">
</colgroup>
<thead>
<tr class="header">
<th style="text-align: left;">dia</th>
<th style="text-align: left;">palco</th>
<th style="text-align: left;">artist</th>
<th style="text-align: left;">hora_ini</th>
<th style="text-align: left;">hora_fim</th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td style="text-align: left;">sabado</td>
<td style="text-align: left;">Palco Skol</td>
<td style="text-align: left;">cage the elephant</td>
<td style="text-align: left;">2017-03-25 16:25:00</td>
<td style="text-align: left;">2017-03-25 17:25:00</td>
</tr>
<tr class="even">
<td style="text-align: left;">sabado</td>
<td style="text-align: left;">Palco Skol</td>
<td style="text-align: left;">metallica</td>
<td style="text-align: left;">2017-03-25 21:00:00</td>
<td style="text-align: left;">2017-03-25 23:00:00</td>
</tr>
<tr class="odd">
<td style="text-align: left;">sabado</td>
<td style="text-align: left;">Palco Skol</td>
<td style="text-align: left;">doctor pheabes</td>
<td style="text-align: left;">2017-03-25 12:05:00</td>
<td style="text-align: left;">2017-03-25 13:05:00</td>
</tr>
<tr class="even">
<td style="text-align: left;">sabado</td>
<td style="text-align: left;">Palco Skol</td>
<td style="text-align: left;">rancid</td>
<td style="text-align: left;">2017-03-25 18:35:00</td>
<td style="text-align: left;">2017-03-25 19:35:00</td>
</tr>
<tr class="odd">
<td style="text-align: left;">sabado</td>
<td style="text-align: left;">Palco Skol</td>
<td style="text-align: left;">suricato</td>
<td style="text-align: left;">2017-03-25 14:15:00</td>
<td style="text-align: left;">2017-03-25 15:15:00</td>
</tr>
<tr class="even">
<td style="text-align: left;">sabado</td>
<td style="text-align: left;">Palco Onix</td>
<td style="text-align: left;">the 1975</td>
<td style="text-align: left;">2017-03-25 17:30:00</td>
<td style="text-align: left;">2017-03-25 18:30:00</td>
</tr>
<tr class="odd">
<td style="text-align: left;">sabado</td>
<td style="text-align: left;">Palco Onix</td>
<td style="text-align: left;">the outs</td>
<td style="text-align: left;">2017-03-25 13:10:00</td>
<td style="text-align: left;">2017-03-25 14:10:00</td>
</tr>
<tr class="even">
<td style="text-align: left;">sabado</td>
<td style="text-align: left;">Palco Onix</td>
<td style="text-align: left;">the xx</td>
<td style="text-align: left;">2017-03-25 19:40:00</td>
<td style="text-align: left;">2017-03-25 20:55:00</td>
</tr>
</tbody>
</table>
</div>
</div>
<p>It’s interesting to note that to build this simple little table we used the <code>httr</code>, <code>rvest</code>, <code>purrr</code>, <code>dplyr</code>, <code>tidyr</code>, <code>lubridate</code>, <code>stringi</code> and <code>forcats</code> packages. Only <code>ggplot2</code> was missing to complete the tidyverse.</p>
<p>PS: 89 fm is not a band, it was just a slot reserved for the radio station’s advertising.</p>
</section>
<section id="extracting-the-popularity-of-the-lollapalooza-bands-on-spotify" class="level3">
<h3 class="anchored" data-anchor-id="extracting-the-popularity-of-the-lollapalooza-bands-on-spotify">Extracting the popularity of the Lollapalooza bands on Spotify</h3>
<p>Now let’s use the <code>Rspotify</code> package to extract the popularity of the bands listed in the <code>lolla2017_programacao</code> data.frame. For that, I used an official Spotify playlist made by the Lollapalooza team itself. This playlist is identified by the id <code>1mHoPn6JpbtWtoBuvSXrVm</code> in the Spotify database.</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb4" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb4-1">meu_token <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">spotifyOAuth</span>(app_id, client_id, client_secret) <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># put your info provided by Spotify here.</span></span></code></pre></div></div>
</div>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb5" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb5-1">lolla2017_playlist <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">getPlaylistSongs</span>(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"lollabr"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"1mHoPn6JpbtWtoBuvSXrVm"</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">token =</span> meu_token) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb5-2">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">mutate</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">artistInfo =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">map</span>(artistId, getArtistinfo),</span>
<span id="cb5-3">         <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">artist =</span> artist <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span> tolower) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb5-4">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">rename</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">track_popularity =</span> popularity,</span>
<span id="cb5-5">         <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">track_id =</span> id) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb5-6">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">unnest</span>(artistInfo) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb5-7">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">select</span>(artist, id, name, popularity, followers)</span></code></pre></div></div>
</div>
<p>Some bands were left out of the playlist, so I searched for the artist name in the Spotify API itself to retrieve the corresponding id. The function that does this is <code>searchArtist()</code>.</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb6" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb6-1"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># retrieving info on the artists the playlist forgot ----------------------</span></span>
<span id="cb6-2"></span>
<span id="cb6-3">possibly_searchArtist <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">possibly</span>(searchArtist, <span class="cn" style="color: #8f5902;
background-color: null;
font-style: inherit;">NA_character_</span>)</span>
<span id="cb6-4"></span>
<span id="cb6-5">artistas_fora_da_playlist <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> lolla2017_programacao <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb6-6">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">filter</span>(<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">!</span>artist <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%in%</span> lolla2017_playlist<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span>artist) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%$%</span></span>
<span id="cb6-7">  artist <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb6-8">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">data_frame</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">artist =</span> .) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb6-9">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">mutate</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">search_artist =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">map</span>(artist, <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">~</span> .x <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span> possibly_searchArtist),</span>
<span id="cb6-10">         <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">artist_info =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">map2</span>(search_artist, artist, <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">~</span> {</span>
<span id="cb6-11">           <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">if</span>(.x <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span> is.na) {<span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">data.frame</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">search_artist =</span> <span class="cn" style="color: #8f5902;
background-color: null;
font-style: inherit;">NA</span>)} <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">else</span> {</span>
<span id="cb6-12">           .x <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb6-13">             <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">mutate</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">name =</span> name <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span> tolower) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb6-14">             <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">filter</span>(name <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%in%</span> .y) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb6-15">             <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">head</span>(<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span>)</span>
<span id="cb6-16">         }})) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb6-17">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">select</span>(<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">-</span>search_artist) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb6-18">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">unnest</span>(artist_info) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb6-19">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">select</span>(artist, id, name, popularity, followers)</span></code></pre></div></div>
</div>
<p><strong>Result</strong></p>
<div class="cell">
<div class="cell-output-display">
<table class="caption-top table table-sm table-striped small">
<colgroup>
<col style="width: 21%">
<col style="width: 30%">
<col style="width: 21%">
<col style="width: 14%">
<col style="width: 13%">
</colgroup>
<thead>
<tr class="header">
<th style="text-align: left;">artist</th>
<th style="text-align: left;">id</th>
<th style="text-align: left;">name</th>
<th style="text-align: right;">popularity</th>
<th style="text-align: right;">followers</th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td style="text-align: left;">ricci</td>
<td style="text-align: left;">1EUMh6DZo2CfpolG75YQBL</td>
<td style="text-align: left;">ricci</td>
<td style="text-align: right;">49</td>
<td style="text-align: right;">6116</td>
</tr>
<tr class="even">
<td style="text-align: left;">jimmy eat world</td>
<td style="text-align: left;">3Ayl7mCk0nScecqOzvNp6s</td>
<td style="text-align: left;">jimmy eat world</td>
<td style="text-align: right;">66</td>
<td style="text-align: right;">361785</td>
</tr>
<tr class="odd">
<td style="text-align: left;">89 fm</td>
<td style="text-align: left;">NA</td>
<td style="text-align: left;">NA</td>
<td style="text-align: right;">NA</td>
<td style="text-align: right;">NA</td>
</tr>
<tr class="even">
<td style="text-align: left;">martin garrix</td>
<td style="text-align: left;">60d24wfXkVzDSfLS6hyCjZ</td>
<td style="text-align: left;">martin garrix</td>
<td style="text-align: right;">85</td>
<td style="text-align: right;">2244761</td>
</tr>
<tr class="odd">
<td style="text-align: left;">illusionize</td>
<td style="text-align: left;">3RloA7E4XMItSP4FjMBv3L</td>
<td style="text-align: left;">illusionize</td>
<td style="text-align: right;">46</td>
<td style="text-align: right;">30054</td>
</tr>
</tbody>
</table>
</div>
</div>
</section>
<section id="putting-it-all-together" class="level3">
<h3 class="anchored" data-anchor-id="putting-it-all-together">Putting it all together</h3>
<p>Now let’s join the Lolla schedule with the Spotify info. The key is <code>artist</code>.</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb7" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb7-1">lolla2017 <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">left_join</span>(lolla2017_programacao,</span>
<span id="cb7-2">                       lolla2017_playlist <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">bind_rows</span>(artistas_fora_da_playlist),</span>
<span id="cb7-3">                       <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">by =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"artist"</span>) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb7-4">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">select</span>(<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">-</span>id, <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">-</span>name) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb7-5">  dplyr<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">::</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">filter</span>(followers <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span> is.na <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span> not)</span></code></pre></div></div>
</div>
<p><strong>Final dataset</strong></p>
<div class="cell">
<div class="cell-output-display">
<table class="caption-top table table-sm table-striped small">
<colgroup>
<col style="width: 7%">
<col style="width: 11%">
<col style="width: 18%">
<col style="width: 20%">
<col style="width: 20%">
<col style="width: 11%">
<col style="width: 10%">
</colgroup>
<thead>
<tr class="header">
<th style="text-align: left;">dia</th>
<th style="text-align: left;">palco</th>
<th style="text-align: left;">artist</th>
<th style="text-align: left;">hora_ini</th>
<th style="text-align: left;">hora_fim</th>
<th style="text-align: right;">popularity</th>
<th style="text-align: right;">followers</th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td style="text-align: left;">sabado</td>
<td style="text-align: left;">Palco Skol</td>
<td style="text-align: left;">cage the elephant</td>
<td style="text-align: left;">2017-03-25 16:25:00</td>
<td style="text-align: left;">2017-03-25 17:25:00</td>
<td style="text-align: right;">72</td>
<td style="text-align: right;">745453</td>
</tr>
<tr class="even">
<td style="text-align: left;">sabado</td>
<td style="text-align: left;">Palco Skol</td>
<td style="text-align: left;">metallica</td>
<td style="text-align: left;">2017-03-25 21:00:00</td>
<td style="text-align: left;">2017-03-25 23:00:00</td>
<td style="text-align: right;">80</td>
<td style="text-align: right;">3047126</td>
</tr>
<tr class="odd">
<td style="text-align: left;">sabado</td>
<td style="text-align: left;">Palco Skol</td>
<td style="text-align: left;">doctor pheabes</td>
<td style="text-align: left;">2017-03-25 12:05:00</td>
<td style="text-align: left;">2017-03-25 13:05:00</td>
<td style="text-align: right;">20</td>
<td style="text-align: right;">313</td>
</tr>
<tr class="even">
<td style="text-align: left;">sabado</td>
<td style="text-align: left;">Palco Skol</td>
<td style="text-align: left;">rancid</td>
<td style="text-align: left;">2017-03-25 18:35:00</td>
<td style="text-align: left;">2017-03-25 19:35:00</td>
<td style="text-align: right;">60</td>
<td style="text-align: right;">220182</td>
</tr>
<tr class="odd">
<td style="text-align: left;">sabado</td>
<td style="text-align: left;">Palco Skol</td>
<td style="text-align: left;">suricato</td>
<td style="text-align: left;">2017-03-25 14:15:00</td>
<td style="text-align: left;">2017-03-25 15:15:00</td>
<td style="text-align: right;">41</td>
<td style="text-align: right;">52326</td>
</tr>
<tr class="even">
<td style="text-align: left;">sabado</td>
<td style="text-align: left;">Palco Onix</td>
<td style="text-align: left;">the 1975</td>
<td style="text-align: left;">2017-03-25 17:30:00</td>
<td style="text-align: left;">2017-03-25 18:30:00</td>
<td style="text-align: right;">77</td>
<td style="text-align: right;">1600845</td>
</tr>
<tr class="odd">
<td style="text-align: left;">sabado</td>
<td style="text-align: left;">Palco Onix</td>
<td style="text-align: left;">the outs</td>
<td style="text-align: left;">2017-03-25 13:10:00</td>
<td style="text-align: left;">2017-03-25 14:10:00</td>
<td style="text-align: right;">28</td>
<td style="text-align: right;">3788</td>
</tr>
<tr class="even">
<td style="text-align: left;">sabado</td>
<td style="text-align: left;">Palco Onix</td>
<td style="text-align: left;">the xx</td>
<td style="text-align: left;">2017-03-25 19:40:00</td>
<td style="text-align: left;">2017-03-25 20:55:00</td>
<td style="text-align: right;">76</td>
<td style="text-align: right;">2383923</td>
</tr>
</tbody>
</table>
</div>
</div>
</section>
</section>
<section id="results" class="level2">
<h2 class="anchored" data-anchor-id="results">Results</h2>
<section id="hypothesis-i" class="level3">
<h3 class="anchored" data-anchor-id="hypothesis-i">Hypothesis I</h3>
<blockquote class="blockquote">
<p><strong>Hypothesis I:</strong> the organizers used the strategy of spreading band popularity evenly throughout the day.</p>
</blockquote>
<p>One of the many logistical challenges of the event is allocating the bands to time slots across the four different stages.</p>
<p>I went to the event on Saturday and heard that <em>Cage The Elephant</em> had been one of the first bands to play. I knew how popular the band was (according to Spotify, more popular than <em>The Strokes</em>) and right away I found it odd that the event had them play so early.</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb8" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb8-1">lolla2017_grafico <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> lolla2017 <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb8-2">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">mutate</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">hora =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">map2</span>(hora_ini, hora_fim, <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">~</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">seq</span>(.x, .y, <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">30</span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">*</span><span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">60</span>) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">floor_date</span>(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"30 minutes"</span>))) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb8-3">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">unnest</span>(hora) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb8-4">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">group_by</span>(dia, hora, palco) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb8-5">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">summarise</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">artist =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">first</span>(artist),</span>
<span id="cb8-6">            <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">n =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">n</span>(),</span>
<span id="cb8-7">            <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">mean_popularity =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">mean</span>(popularity))</span>
<span id="cb8-8"></span>
<span id="cb8-9">lolla2017_grafico <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb8-10">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">ggplot</span>(<span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">aes</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">x =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">ymd_hm</span>(<span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">format</span>(hora, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"2017-03-26 %H%M"</span>)), <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">y =</span> mean_popularity, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">colour =</span> palco)) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span></span>
<span id="cb8-11">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">geom_line</span>() <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span></span>
<span id="cb8-12">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">geom_point</span>() <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span></span>
<span id="cb8-13">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">geom_point</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">data =</span> lolla2017_grafico <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">filter</span>(artist <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%in%</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"cage the elephant"</span>), <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">colour =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"red"</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">size =</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">2</span>) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span></span>
<span id="cb8-14">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">geom_text</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">data =</span> lolla2017_grafico <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">filter</span>(artist <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%in%</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"cage the elephant"</span>) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">head</span>(<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span>), <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">aes</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">label =</span> artist), <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">colour =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"red"</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">hjust =</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">0</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">vjust =</span> <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">-</span><span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span>) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span></span>
<span id="cb8-15">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">facet_wrap</span>(<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">~</span>dia) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span></span>
<span id="cb8-16">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">labs</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">x =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Hora do dia"</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">y =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Popularidade média"</span>) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span></span>
<span id="cb8-17">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">theme</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">text =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">element_text</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">size =</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">16</span>))</span></code></pre></div></div>
<div class="cell-output-display">
<div>
<figure class="figure">
<p><img src="https://athospd.github.io/athosdamiani/posts/2017-03-27-lollapalooza-sp-2017-segundo-spotify/index_files/figure-html/unnamed-chunk-12-1.png" class="img-fluid figure-img" width="864"></p>
</figure>
</div>
</div>
</div>
<p>The plot above agrees with the common sense that the popular ones are left for the end, which doesn’t help confirm the hypothesis that <em>Cage The Elefant</em> was badly placed.</p>
</section>
<section id="hypothesis-ii" class="level3">
<h3 class="anchored" data-anchor-id="hypothesis-ii">Hypothesis II</h3>
<blockquote class="blockquote">
<p><strong>Hypothesis II:</strong> in terms of band popularity, Sunday was better than Saturday.</p>
</blockquote>
<p>Talking with friends and acquaintances, I noticed that most of them were either going on Sunday or would rather go on Sunday if they had the chance. This made me wonder whether there really was a higher concentration of good bands on Sunday.</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb9" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb9-1"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">ggplot</span>(lolla2017 <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb9-2">         <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">mutate</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">artist =</span> artist <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">fct_reorder</span>(popularity, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">.desc =</span> <span class="cn" style="color: #8f5902;
background-color: null;
font-style: inherit;">TRUE</span>))) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span></span>
<span id="cb9-3">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">geom_bar</span>(<span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">aes</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">x =</span> artist, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">y =</span> popularity, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">fill =</span> dia), <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">stat =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"identity"</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">position =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"dodge"</span>) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span></span>
<span id="cb9-4">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">theme</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">text =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">element_text</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">size =</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">16</span>),</span>
<span id="cb9-5">        <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">axis.text.x =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">element_text</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">angle =</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">90</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">hjust =</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">vjust =</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0.4</span>))</span></code></pre></div></div>
<div class="cell-output-display">
<div>
<figure class="figure">
<p><img src="https://athospd.github.io/athosdamiani/posts/2017-03-27-lollapalooza-sp-2017-segundo-spotify/index_files/figure-html/unnamed-chunk-13-1.png" class="img-fluid figure-img" width="864"></p>
</figure>
</div>
</div>
</div>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb10" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb10-1"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">ggplot</span>(lolla2017) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span></span>
<span id="cb10-2"> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">geom_density</span>(<span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">aes</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">fill =</span> dia, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">x =</span> popularity, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">colour =</span> dia), <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">fill =</span> <span class="cn" style="color: #8f5902;
background-color: null;
font-style: inherit;">NA</span>) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span></span>
<span id="cb10-3">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">theme</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">text =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">element_text</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">size =</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">16</span>),</span>
<span id="cb10-4">        <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">axis.text.x =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">element_text</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">angle =</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">90</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">hjust =</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">vjust =</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0.4</span>))</span></code></pre></div></div>
<div class="cell-output-display">
<div>
<figure class="figure">
<p><img src="https://athospd.github.io/athosdamiani/posts/2017-03-27-lollapalooza-sp-2017-segundo-spotify/index_files/figure-html/unnamed-chunk-14-1.png" class="img-fluid figure-img" width="864"></p>
</figure>
</div>
</div>
</div>
<p>Conclusion: nothing suggests there was an imbalance. I think my circle of friends has some strange bias.</p>
</section>
<section id="hypothesis-iii" class="level3">
<h3 class="anchored" data-anchor-id="hypothesis-iii">Hypothesis III</h3>
<blockquote class="blockquote">
<p><strong>Hypothesis III:</strong> band popularity was balanced across the different stages.</p>
</blockquote>
<p>When I questioned hypothesis I, I also thought about how hard it is to place the bands on the different stages. Since thousands of people would be fighting for space, it would be in the organizers’ interest to keep them as spread out as possible for several reasons: better queue flow, more comfort, less risk of accidents, among others, and a good way to do that would be to make the stages equally attractive so there wouldn’t be a big crowd in a single spot.</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb11" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb11-1"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">ggplot</span>(lolla2017 <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span id="cb11-2">         <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">mutate</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">palco =</span> palco <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span> as.factor <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">fct_reorder</span>(popularity, mean))) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span></span>
<span id="cb11-3">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">geom_boxplot</span>(<span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">aes</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">fill =</span> palco, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">y =</span> popularity, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">x =</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span>)) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span></span>
<span id="cb11-4">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">theme</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">text =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">element_text</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">size =</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">16</span>),</span>
<span id="cb11-5">        <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">axis.text.x =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">element_blank</span>()) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span></span>
<span id="cb11-6">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">labs</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">x =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">""</span>)</span></code></pre></div></div>
<div class="cell-output-display">
<div>
<figure class="figure">
<p><img src="https://athospd.github.io/athosdamiani/posts/2017-03-27-lollapalooza-sp-2017-segundo-spotify/index_files/figure-html/unnamed-chunk-15-1.png" class="img-fluid figure-img" width="864"></p>
</figure>
</div>
</div>
</div>
<p>The Skol stage had the least variation in popularity and usually had artists of medium to high popularity, but the AXE and Onix stages got the heavyweight artists. The Perry’s stage hosted the most lesser-known artists.</p>
</section>
</section>
<section id="final-remarks" class="level2">
<h2 class="anchored" data-anchor-id="final-remarks">Final remarks</h2>
<p>The topic covered here wasn’t useful, I agree, but we went through almost every step of a typical data analysis process. We did web scraping, used APIs, tidied the data, structured the information, created variables and made plots. Only the modeling part was left out. And it’s no coincidence that all the tidyverse packages were useful in this work.</p>
<p>The homework is to find some interesting data on the internet and apply the steps we learned here!</p>


</section>

 ]]></description>
  <category>analysis</category>
  <category>spotify</category>
  <category>lollapalooza</category>
  <category>api</category>
  <guid>https://athospd.github.io/athosdamiani/posts/2017-03-27-lollapalooza-sp-2017-segundo-spotify/</guid>
  <pubDate>Mon, 27 Mar 2017 00:00:00 GMT</pubDate>
  <media:content url="https://athospd.github.io/athosdamiani/posts/2017-03-27-lollapalooza-sp-2017-segundo-spotify/lolla-spotify_s.webp" medium="image" type="image/webp"/>
</item>
</channel>
</rss>
