Esp8266 WebServer Farm - Again

After several long months, I have reactivated my Esp8266 WebServer Farm.

Use this links to all of my Esp8366 project blog posts.

Esp8266 WebServer Farm

Currently, one of my WebServers can be accessed as: http://esp.wa0uwh.com:8154

The main reason that it has be off-line, is lack of interest, and recently, I could only get the code to compile on the old Arduino IDE (Rev 1.6.x).  The code is quite large and I took advantage of many "tabs" for code fragments, which are concatenated before presented to the compiler.  The new Arduino IDE Revision (1.8.1) is much more strict regarding Header Files, code files, and program structure. There were just TOO MANY errors to correct to keep my interest in solving the problem.

Once started to solve the problems and not getting much accomplished, my Friend Jeff - Ko7m, suggest another IDE, he suggested "PlatformIO IDE".

With little work, I found and installed PlatformIO, but alas, I found similar problems and concerns with it.  But little-by-little after much on-line reading, Jeff and I worked out the best file structures that worked with PlatformIO.

I have not moved the new code structure to GitHub, but maybe soon.


-- Home Page: https://WA0UWH.blogspot.com

Simple Complexity - By Proxy

UPDATED:
Added Details and Links

For sometime I have struggled with how to allow public access to my home web servers and Esp8266 modules without opening up my network to abuse. In the past I have managed public access via my Router, by changing its configuration of Port Forwarding and NAT.  This works, but it is a pain to manage and generally requiring a re-boot of the network for each change.

Recently, I have discovered (actually re-discovered) that an Apache2 Web Proxy Server is much easier to manage, but it has a bit of a steep learning curve, with a lot of manual pages to read. One key concept is that Apache2 ReWriteRules are a super-set of the functionality of ProxyPass, each have their own documentation web pages.

After building the Required Config files

Now, on my Router I allow only Ports: http 80, 8040, 8160, and a private ssh port for access from the Internet. Ports 8040, and 8160 are still open for historical reasons, that is, they are used for my published Web Pages at: http://www.WA0UWH.com:8040, and my Esp8266 Server Farm devices.

The Apache2 Web Server supports: Virtual Host Names with Proxy Redirects, ReWriteRules, and ProxPass. By setting up "*.wa0uwh.com" as a CNAME (an aliases) to "www.wa0uwh.com" at my DNS Provider, I can use any "device name" I would like in the config files to initiate a proxy process. For example: I can now use and publish "http://node129.wa0uwh.com" for one of my Esp8266 Web Server modules. The actual connection details and security are all hidden behind the proxy curtains.

The normal web page port 80 is setup with a default virtual host page of; "Error 404", only configured virtual hosts and named devices are let through the proxy.

Note: The service and/or host that is selected is a combination of both Port Number and Host Name (or alias). For an incoming connection, the file is scanned from top to bottom, only the first match is used to select the service.

The following are excerpts from my Apache2 Default Virtual Host configuration file.

ServerName default.wa0uwh.com
LogLevel alert rewrite:trace1

<VirtualHost *:80>

    ServerAdmin webmaster@localhost
    DocumentRoot /var/www/DEFAULT/Public

    ErrorLog ${APACHE_LOG_DIR}/error.log
    CustomLog ${APACHE_LOG_DIR}/access.log combined

</VirtualHost>

# End

The following are excerpts from my Apache2 Named Virtual Host configuration file.

## Main Web Pages

## Loft Raspberry PI
<VirtualHost *:8040 *:80 >

    ServerName  rp21.wa0uwh.com
        RewriteEngine on
        ServerSignature Off
        RewriteRule /(.*)$      http://192.168.___.___/$1 [P,L]

</VirtualHost

## Loft Raspberry PI
<VirtualHost *:8040 *:80 >

    ServerName  rp22.wa0uwh.com
        RewriteEngine on
        ServerSignature Off
        RewriteRule /(.*)$      http://192.168.___.___/$1 [P,L]

</VirtualHost>

# Esp8266 Node on Published Port 8160
<VirtualHost *:8160 >

    ServerName  node.wa0uwh.com
    ServerAlias node*.wa0uwh.*
    ServerAlias localhost
        RewriteEngine on
        RewriteRule /(.*)$    http://192.168.___.___/$1 [P,L]

</VirtualHost>

## Loft Esp8266 Nodes
<VirtualHost *:8040 *:80 >

    ServerName  node.wa0uwh.com
    ServerAlias node*.wa0uwh.com
        RewriteEngine on
        ServerSignature Off
        RewriteCond %{HTTP_HOST} ^node(129|162|164|168|169|170|172)\.wa0uwh\.com [NC]
        RewriteRule /(.*)$      http://192.168.___.%1/$1 [P,L]
        RewriteRule /(.*)$ -  [R=404,L]

</VirtualHost>


## Loft WA0UWH Web Server
<VirtualHost *:8040 *:80>

    ServerName www.wa0uwh.com
    ServerAlias *.wa0uwh.com
    ServerAlias localhost 192.168.__.__ 
        DocumentRoot    /var/www/WA0UWH/Public
        Alias /gallery  /var/www/WA0UWH/Public/Gallery

</VirtualHost>

# End

Note, the above is just an excerpt from my Apache2 Virtual Host config file. For security reasons, the details; actual IPA's (___), BlackListing, Hacker Traps, Web Abuse Traps, and HoneyPots, are NOT included . Google is your friend for suggested configurations.

Now, with simple edits of the Apache2 Virtual Host config file, I can turn ON or OFF, devices and/or services as desired, while leaving only the http and ssh ports open for public access at the router.

Also note: each of my Raspberry PI Web Servers also have similar Apache2 Virtual Host config files, that is: Proxies are serving Proxies, and most often the actual destination is at different physical locations, and on different Networks! All unseen for my Internet users.


-- Home Page: https://WA0UWH.blogspot.com

Esp8266 - Interactive SMS from the Web Page

UPDATE: Mon Nov 30 21:34:56 PST 2015
Added Links and Detail
Added RAW HTML listing

In the previous post I listed a code snippet that provides non-interactive SMS from the Esp8266. But sometime it is desirable to provide a simple interactive SMS interface for the Web User. An interactive SMS interface could be used to "page the SysAdmin" with the web user's supplied message.

Example Interactive Web SMS Interface

The interactive SMS interface is simple and requires only some HTML code to be inserted within a web page. Note: the previous post was a SMS method where user interaction was not needed nor desired.

Note: the Open SMS Gateway domain name that is used in the example is "textbelt.com", read their web page for details for Canadian and International use.

The following are excerpts from my ERB-EspWebServer.

Note: the RAW HTML code could be extracted from the example below and used with any Web Server.

The Setup:

// The Setup

    #define OPT_MONITOR_DEBUG_OUTPUT 1
   
    // ERB - Force format stings and string constants into FLASH Memory
    // I use E() as an Alias for F() so that DEBUG code can be inserted
    #define  E(x) F(x)
    // Used as an F() when being used as the first Element
    // of a Multi-Element Expression
    #define sE(x) String( F(x) )


// Defaults for Pager
    #define DEFAULT_PAGER_GATEWAY "textbelt.com"
    #define DEFAULT_PAGER_NUMBER  "202-555-1212"

    #define SYSTEM_ADMIN_PAGER_NUMBER DEFAULT_PAGER_NUMBER

Web Server Code Arduino Sketch Snippet:

    // Pager - User Interactive Web Interface
    sz += wprintln( E("\r\n<!-- Pager - User Interactive Web Interface -->") );
    sz += wprintln( E("<center>") );
    sz += wprintln( E("<hr width='75%' />") );

    sz += wprintln(sE("<form action="http://")
          + DEFAULT_PAGER_GATEWAY + E("/text method='POST' target='SMS_Report' >") );
       sz += wprintln( E("  <b>Page SysAdmin with </b>") );
       sz += wprintln(sE("  <input name='number' type='hidden' value='")
             + SYSTEM_ADMIN_PAGER_NUMBER + E("' />") );
       sz += wprintln( E("  <b>Message:</b>") );
       sz += wprintln(sE("  <input name='message' type='text' value='")
             + gDeviceName + E(": ' />") );
       sz += wprintln( E("  <input type='submit' value='GO' />") );
    sz += wprintln( E("</form>") );
    
    sz += wprintln( E("Or<br>") );
    
    sz += wprintln(sE("<form action="http://")
          + DEFAULT_PAGER_GATEWAY + E("/text method='POST' target='SMS_Report' >") );
       sz += wprintln( E("  <b>SMS Number:</b>") );
       sz += wprintln( E("  <input name='number' type='tel' />") );
       sz += wprintln( E("  <b>Message:</b>") );
       sz += wprintln(sE("  <input name='message' type='text' value='")
             + gDeviceName + E(": ' />") );
       sz += wprintln( E("  <input type='submit' value='GO' />") );
    sz += wprintln( E("</form>") );

    sz += wprintln( E("</center>") );

Note: I use my own custom "wprintln" function within the above example, modify as necessary.

For reference and to make things simple, I have extracted just the HTML from the above Sketch, as shown below.

HTML:

<!-- Pager - User Interactive Web Interface -->
<center>
<hr width='75%'>

<form method='POST' action='http://textbelt.com/text' target='SMS_Report' >
  <b>Page SysAdmin with </b>
  <input type='hidden' name='number' value='202-555-1212' />
  <b>Message:</b>
  <input type='text' name='message' value='Nod129: ' />
  <input type='submit' value='GO' />
</form>

Or<br>

<form method='POST' action='http://textbelt.com/text' target='SMS_Report' >
  <b>SMS Number:</b>
  <input type='tel' name='number' />
  <b>Message:</b>
  <input type='text' name='message' value='Nod129: ' />
  <input type='submit' value='GO' />
</form>

</center>

If you find this useful, leave a comment.

-- Home Page: https://WA0UWH.blogspot.com
 

Esp8266 - with SMS

UPDATE: Sun Nov 29 22:14:04 PST 2015
Added details and Links


For my Esp8266 I wanted a easy way to send to my Cell Phone an SMS message.

For example: On a normal computers (Workstations and Raspberry PI) it is a simple mater of using "curl" to access an Open SMS Gateway (see previous post). I used this feature to report simple things like "reboot" and "backup finished" messages.

One such Open SMS Gateway is "textbelt.com", which is the one that I use. But, recently they were "black listed" by some of the Phone Carriers (my carrier) for abuse, apparently there was too much illegal traffic by the Gateway's users.

Textbelt has recently fixed the abuse and their Gateways is up and running again, or at least for my carrier.

For the Exp8266, I have created some functions that implements the "curl" protocol, (the heavy lifting), so now paging to my Cell Phone is a snap.

I am providing the functions here as just application snippets, hopefully you find them useful. I will put this in my ERB-EspWebServer GitHub (eventually).

The Setup:

// The Setup

    #define OPT_MONITOR_DEBUG_OUTPUT 1
   
    // ERB - Force format stings and string constants into FLASH Memory
    // USE WITH CAUTION !
    // I use E() as an Alias for F() so that DEBUG code can be inserted
    #define  E(x) F(x)
    // Used as an F() when being used as the first Element of a Multi-Element Expression
    #define sE(x) String( F(x) )

// Defaults for Pager
    #define DEFAULT_PAGE_GATEWAY "textbelt.com"
    #define DEFAULT_PAGE_PORT     80
    #define DEFAULT_PAGE_NUMBER  "202-555-1212"
    #define DEFAULT_PAGE_MESSAGE "Test"

    boolean gPageStnReady = false;

The Functions:

// Pager

// ###########################################################
//////////////////////////////////////////////////////////////
//
// Provides easy access to page via SMS Gateway
//
boolean
//ICACHE_FLASH_ATTR
page( String aMesgText = DEFAULT_PAGE_MESSAGE,
      String aNumber   = DEFAULT_PAGE_NUMBER,
      String aHost     = DEFAULT_PAGE_GATEWAY,
         int aPort     = DEFAULT_PAGE_PORT )
{
    
    digitalWrite ( gBluLED, ON );
    
        #if (OPT_MONITOR_DEBUG_OUTPUT) > 0
            Serial.println ( );
            Serial.println ( E("Starting Pager Client ...") );
        #endif
        
        String timeUTCBuf = "";
        
        String mesgTextHeader = String(gDeviceName)
            + E(": ")
            + String(gRev)
            + E(", ")
            + timeUTC(timeUTCBuf, gTimeZONE)
            + E(" - ");
    
        String mesg="number=" + aNumber
               + "&message=" + mesgTextHeader + aMesgText ;
    
        #if (OPT_MONITOR_DEBUG_OUTPUT) > 0
            Serial.println( sE("  Connecting to: ") + aHost );
        #endif
      
        // Use WiFiClient class to create TCP connections
        WiFiClient client;
        if (!client.connect(aHost.c_str(), aPort)) {
          #if (OPT_MONITOR_DEBUG_OUTPUT) > 0
              Serial.println("  Connection Failed");
          #endif
          return false;
        }
        
        #if (OPT_MONITOR_DEBUG_OUTPUT) > 0 
            Serial.println( sE("  MESG: ") + mesg );
        #endif
   
        // This will send the request to the server
        client.print( sE("POST /text/ HTTP/1.1\r\n")
                     + E("User-Agent: Esp8266/") + gRev + E("\r\n")
                     + E("Host: ") + aHost + E("\r\n")
                     + E("Accept: */*\r\n")
                     + E("Content-Length: ")
                     + String( mesg.length() ) + E("\r\n")
                     + E("Content-Type: application/x-www-form-urlencoded\r\n")
                     + E("\r\n")  // The Required Blank Line
                     + mesg + E("\r\n") );
    
        delay(10);

        // Wait for Client to Start Sending
        int i = 0;
        while( !client.available() ) {
          delay(10);
          if (i++ > 200) break;
        }
          
        // Read all the lines of the reply from server and print some to Serial
        boolean myStatus = false;
        while(client.available()){
          String line = client.readStringUntil('\n');
          if(line.indexOf("\"success\": true") >= 0) {
              myStatus = true;
          }
          #if (OPT_MONITOR_DEBUG_OUTPUT) > 0 
              if(line.indexOf("\"success\": ") >= 0 ||
                 line.indexOf("\"message\": ") >= 0
              ) Serial.println(line);
          #endif
        }
        
        client.flush();
        client.stop();
        #if (OPT_MONITOR_DEBUG_OUTPUT) > 0 
            Serial.println("  Connection Closed");
        #endif
         
    digitalWrite ( gBluLED, OFF );

    return myStatus;
  
}


// ###########################################################
//////////////////////////////////////////////////////////////
//
// Provides easy access to Pager via SMS Gateway
//
unsigned long gPageHoldOff = millis();
void
//ICACHE_FLASH_ATTR
pageOnStnReady() {
    
      if ( gPageHoldOff < millis() ) {
        
          gPageHoldOff = millis() + 1 * MINs;
        
          if ( gPageStnReady == false ) {
               gPageStnReady = page("Stn Ready");
          }
      }
  
    return;
  
}

// ###########################################################
// End

Let me know if you find this useful, leave a comment.

-- Home Page: https://WA0UWH.blogspot.com

Esp8266 Esp-13 - Testing

Today I found time to do some Contrived Stress and Speed Tests on my latest ERB-EspWebServer Code, using the Esp8266 Esp-13 module.

• For the Tests , the Esp-13 was on a local WIFI network, running at 160MHz, with 4MB of 80 MHz Flash.
• My Workstation is hard wired connected to the same network via a switch.
• The Http Transfer Protocol used is "CHUNKED".
• The Automatically Refreshed (4 Second) Home Page was a little over 30KB.
• My Workstation had 20 Tabs open, each connected to the Esp's Home Page.
• The included Graphic was showing a steady 15.8KB FreeHeap.
• The Test Home Page has; Text, Graphs, Charts, and SVG Images, all Dynamically Created for each Http request.
• Also, during the Test, mDNS was scheduled and updated each 5 minutes, ntp was updated each 30 minutes.
• Diagnostic Trace and Status was being Created and Displayed on the Workstation Arduino IDE Monitor via USB.
• My Normal Workstation work (which was unavoidable) was also being done during the Tests.

If I have my maths , data and methods correct, the results is:
( 20Esp Tabs x 30KB ) / 4Sec = 150KB Per Second Created and Transferred

One of 4 Groups of 5 Esp Tabs

The Esp-13 ran for about 1 hour, before I got GREEDY, an attempted to open more Browser Tabs. At about 35 Tabs things started to FAIL, the update page rate was decreasing, and eventually the Esp-13 crashed and rebooted itself.

Typical Monitor Output

That was fun, . . . the Esp-13 was just . . . Smok'ng !!
I am impressed and very happy with the results.

I will soon update my GitHub Repository with the revised and much faster code.

Let me know if you are interested in participating in a Live Internet Stress Test.

The initial Internet Tests (with my friend Jeff) suggests the Home Page will update at about 6 seconds per page over the Internet, with the Auto Update rate set to 3 Seconds. And, which, of course, is going to be very dependant on my end of a very slow network. So far, we have not done any multi-Tab Test.

Soon, for some more Tests, I plan to connect the Esp-13 to a Very Fast Internet Connection at my Son's house, stay tuned for that.


-- Home Page: https://WA0UWH.blogspot.com

Esp8266 Esp-13 Programming Fixture

UPDATE:
Updated Image with Chip Bypass Cap


Here is my Esp-13 Programming Fixture in Fritzing format.

Esp8266 Esp-13 Programming Fixture

 The Esp-13 and its Adaptor plugs in place of the dual 9-pin headers, see previous blog page.


-- Home Page: https://WA0UWH.blogspot.com

Esp8266 - Futzing with Fritzing

UPDATE:
Updated Image with Chip Bypass Caps


My friend Larry - KB7KMO mentioned Fritzing (CAE) in one of his email the other day.

Several years ago, I looked at Fritzing when is was very beta, it was interesting then, but lacked a lot of what I needed for Experimental Circuits and PCB Hobby Productions.

I updated to the latest Revision (0.9.2b), to take another look.

For what I am doing now with PlugBoards (protoboards) it may help with documentation and project sharing.

Esp8266 Esp-11 Programming Fixture

This is my first real Fritzing Project, it is my Esp8266 Programming Board. Note: the ERB-EspWebServer Farm (see previous posts) will similarly be documented in another Post.

I still need to create some custom parts. For the above Image, I have used "place-holders" images for the Power Source, Esp's and Prop Plug.

So far, Fritzing has been a little frustrating, but that maybe just my leaning curve. You will likely see Fritzing used in some of my future Posts.

---

UPDATE:
To increase brightness of LED, Reduce the value of the LED Resistors (e.g., down to 2.2K).


-- Home Page: https://WA0UWH.blogspot.com

Esp8266 - An Unplanned Stress Test

Saturday (9/5) about 5:00pm, I had an interesting and unexpected surprise, HACKADAY publish a note about my ERB-EspWebServer.

As a result, my Blog and the my ERB-EspWebServer Farm were hammered with Internet requests.

With only one Esp8266 Farm Server working (nod169) at the time (others were offline) the request rate was intense. For the next 16 hours the single Esp8266 served about 1650 page requests for 1050 unique visitors. When "choked" with Internet Requests, the Esp successfully rebooted itself (several times), and I only occasionally had to help with a power-on reboot.

The above shows the first 16 hours of the onslaught.

The next day about 10:00am the Internet Requests were still coming in, I had to leave for the day, and therefore had to leave the Esp to its own fate. It eventually crashed, and therefore it was mostly down for the next 24 hours.

Once I returned, the Internet Requests started up again, with a little less intensity. And now, 4 days later, the Esp is receiving about 10 hits per hour, see current/live stats here. So far, the Esp has handled about 3500 requests.

Today I added new code to serve a local "Server Farm Photo" which is about 30KB, it can be seen on the current Esp Server Info Page.

This has been a fun and exciting Experiment of Esp Web Server Load Survivability, the Esp8266 did well for itself !!


-- Home Page: https://WA0UWH.blogspot.com

My Experimental Exp8266 Web Sever - is on GitHub

UPDATE:
Here are some interesting links for the Esp8266:

• http://tech.scargill.net/esp13/
• http://www.esp8266.com/
• https://www.facebook.com/esp8266wifi
• https://hackaday.com/2015/09/05/esp8266-web-server-farm

---

I have put my “ERB Esp8266 Experimental Web Server” on GitHub, at:

https://github.com/wa0uwh/ERB-EspWebServer

My Esp8266 server is “often/sometimes” online at:

http://dc02.ebcon.com:8160

The SSID and Passwd are blank and need to be set in the Main.ino file

A shared Visitor Map IdKey is also set in Main.ino, if used as is, any access to your server will generate a “Shared Visitor Map”. In Ham Radio terms, this is called a “Reverse Beacon” to a shared service. Of course, you can request your own private IdKey.

I hope you find it useful as a working example. I have tried to include many pages and functions that I plan to use in my other Esp Projects

NOTE: All web page data is purposefully dynamically created by the Esp8266, except the “Server Farm” image on the InfoPage, and the “Visitor Map” function on the HomePage.

More functions and/or pages will be added as time permits, and I will update the GitHub as necessary.

As maybe expected, the Esp is sometimes fickle, reboot may be needed occasionally.

Feedback is always welcome.


-- Home Page: https://WA0UWH.blogspot.com

Esp8266 Esp-13 to DIP Adaptor

UPDATE:
Added Links and Details

I recently received my homebrew Esp8266 Esp-13 to DIP Adaptors from OSHPark.com. With only a little sanding of the edges they were ready to assemble.

I made the adaptor available as a free "shared project" on OSHPark.com site. They are inexpensive at $4.20 for a set of three. Note: I suggest you order 9 or 12 (or more) to make the free shipping more tolerable for the good folks a OSHPark.

Let me know if you find this adaptor board useful.

The Adaptor and Parts

Soldering the two Castellated Edge Connections was much easier than I thought, with a little rosin on each edge, solder wicked into half-round-holes to form a nice solder bridges between the two boards.

SMT Headers Installed

The SMT Headers were easily soldered on the back. They are inexpensive and available on ebay, for example: here.

Ready to Use

The results can be plugged into a Solder-Less Protoboard, or into a 18 Pin DIP socket. For production, this adaptor would not be appropriate, but for experiments and prototypes, it works great.

Inserted into a Solder-Less Protoboard

The Esp8266 Esp-13 has 4M Bytes of Flash, compared to 512K Bytes on the typical lower numbered modules. Esp-13's are available from several suppliers on ebay. for example: here.

Note: Currently, Esp-13 information is a little sparse on the Internet.

Esp8266 Esp-13 Face Plate
(18 x 20mm)

Esp8266 Esp-13 Pinouts

---

UPDATE:

An another Esp-13 Resource link


-- Home Page: https://WA0UWH.blogspot.com

More Esp8266 Web Server Farm

I have my Experimental Esp8266 Web Server Farm up and running (see photo on original post), for your continued review, it is even faster, and will support many more browse connections.

With "AutoUpdate" set to 60 seconds, I have had as many as 25 browser tabs connected to one of my Esp servers. It normally eventually fails, that is, when more than 4 connections are requesting updates simultaneously. But it was fun to watch the mayhem. Sometimes the ESP even recovers on its own.

I have now included a "Visitor Map", so I can see where connections are coming from, see below.

There are three Esp8266's in this "farm", all using the identical source code. And because of that; the displayed Visitor Map will appear Identical on each, that is because they all contain the same "map registration key" which is used by http://www.revolvermaps.com.

As indicated on a previous post, access to my farm is via a generic IPA:Port connection, which is: http://dc02.ebcon.com:8160, which I had previously changed in my router as necessary to externally connect to a particular Esp (local IPA:80).

For the other devices in my Esp Farm (and bench), use PORT: 8162, 8168, 8169 and/or 8172 (note: one of these will also be also connected to the generic 8160 port). Note: one or more may not be working at any point in time, try one of the others.

With only my local tests and a little WIFI activity, so far the small and simple power supply has survived, and actually worked well. I may need to add "more power" with more concurrent user accesses.

An Interesting Idea: If other Esp Programmers with similar Experimental Public Accessible Esp Servers include the same "map registration key" (a short line line of Html code) we could create a "pseudo reverse beacon" that would display to the "world" our intense shared interest in the Esp. On each configured Esp Web Servers, the "Visitor Map" will display all web connections to our "Esp Web Collective". How fun is that!!??

Let me know if you are interested, I will share details (or you can extract the keys from my Esp web Home Page source list).

It's all fun, . . . with the ESP's !!!

For more Esp8266 information on this blog use this Search.


-- Home Page: https://WA0UWH.blogspot.com

My WIFI Transfer Improvement - 160 Times

UPDATE: Aug 25, 2015

I was using an older revision of the Arduino IDE (old 1.6.4) and an old set of Exp Extensions.

With current IDE 1.6.5 and current Esp Ext's the "server.send()", "server.sendContent()", and "server.client().print()" functions all have about the same performance - fast.

The moral of the story, make sure your environment is up-to-date.

---

I use, and like, the Arduino IDE for ESP8266 software development and have used the "example code" for my initial efforts, but now, my interest expand beyond trivial examples.

My Experimental Web Server Farm

Above is My Experimental Web Server Farm :-)  (a little under powered, but it works)

Like many of the published ESP examples, I used a form of "server.send()" to transfer data via WIFI, but then I found "that" function is somewhat limited (or at least the way I understand it) and therefore I explored the available documentation and source code for other capabilities.

I found "server.sendContent()" which appeared more flexible and more to my desires for a transfer function. I have used "it" for several months for building my apps. My typical transfers are about 5K to 15K bytes, but can be as large as 100K bytes.

My dynamically created WEB Pages typically take about "6.5 seconds" to transfer to a browser, for a small device like the ESP's, I thought that was a little slow, but still usable.

Actually, for my application, I started considering abandoning the ESP's, due to lack of WIFI transfer speed.

BUT THEN, in some obscure recesses of some online web page, I found the "server.client().print()" function - A BIG SPEED IMPROVEMENT. Now the same dynamic WEB Page transfers in only "0.043 seconds" !!!!

For my application, that is a BIG speed improvement of 160 times faster. From my perspective, the little ESP is now a WIFI "screamer" !!

I really like it, . . . when something WORKS !!

This note is published here in hope that someone else finds this information useful. If previously known, it would have save me a lot of time and anguish.

I only wished I understood or knew this earlier, but then I am still learning the ESP (there is much to know). I am sure there are other ESP improvements or coding strategies that I have yet to discover. I plan to publish my "example" application code for others to review, use, and/or abuse.

For your review, my experimental ESP Web Server is (sometimes) running at:

 http://dc02.ebcon.com:8160/

-- Home Page: https://WA0UWH.blogspot.com

Another ESP Adaptor

UPDATE: Here is some important info, watch this thread for progress:

http://www.esp8266.com/viewtopic.php?f=32&t=4094

Yes, this is yet another Esp8266 Adaptor, but this one will support an Esp-12E with its extra six pins. The side pins match the Castellated pins on the Esp-12E. Note: this Diptrace 3D rendering does not know how to correctly represent Castellated connectors.

Top Side - Esp8266 Esp-12E Adaptor

I just sent the design into OshPark.com for manufacture, the boards should be returned in about 2 weeks. The cost is $1.50 each. If all goes well, I will share the design as a Project on OshPark.

Bottom Side - Esp8266 Esp-12E Adaptor

"If", I did everything right (with no mistakes),  this should fit a 22 pin DIP socket, or a normal Protoboard with lots of available holes for jumper wires. The underside used two single row SMT 0.1 inch Headers which are available on Ebay, for example: here.

While designing this adaptor, I considered adding a few Pullup Resistors, Program and Reset switches. But, for the boards size that would require a four layer board, and would decreases Pin flexibility.

I think this adaptor will be my favourite, for small Esp8266 Protoboard development projects. The Esp-12E provides lots of I/O and is still very small (I like that).

Note: this design is the result of experimenting with a "Back-to-Back Configuration" as suggested in a previous post.


Update:




-- Home Page: https://WA0UWH.blogspot.com

Kolban’s book on the ESP8266

I found a wonderful new published ebook and/or PDF for the Esp8266, the link saved here for future reference:

http://neilkolban.com/tech/esp8266/

Also, see:

http://www.esp8266.com/viewtopic.php?f=5&t=4326

Thanks, Kolban for your efforts !


-- Home Page: https://WA0UWH.blogspot.com

Esp8266-Esp12 Adaptors

UPDATE:
See below

I purchased several Esp8266 Esp-12 modules with Adaptor boards, but found that the Adaptor boards are not very Protoboard friendly, that is, when put on a protoboard there are NOT any open (uncovered) hole locations for Jumper Pins. As shown below.

Note: All Holes for Jumpers are Covered by the White Board

And besides, I wanted to create my own adaptor, similar to the Esp-11 Adaptor created earlier (see previous post).

This is what I have created for the Esp-12, it has 9 pins with 2mm spacing on one side and 100mils spacing on the other.

I have made this Adaptor Board a Shared Project for your use at OshPark.com, see Link. Let me know if you find it useful.

The Adaptors are very inexpensive, 3 for $1.00, I suggest ordering them in groups of 12 or 24. The larger count makes the postage and PayPal charges more tolerable for OshPark.

Mounting Configurations

One board is used for each side of the Esp-12. They can be used in several configuration (including one that I had not previously thought of, when creating it, more on that later).

The Castellated Edge Connector needs to be cleaned and "de-burred", the manufacturing process leaves fibreglass dust and burs of metal from the inside of the hole that was cut to make the Castellated Edge Connector.

Clean and Match the Castellated Edge Connector

Align and Solder Header Pins

First, Header Pins were Aligned and Solder , they were later pushed into a Protoboard to help align the Esp-12 on top. Note: the Header is NOT soldered through the holes in this configuration.

All Finished

Another configuration uses the Adaptor parallel with the Esp-12 and with the Header soldered in the holes as normal.

The Flat Wing Configuration

I do not care for this parallel Flat-Wing Configuration, for the same reason that I do not like the original purchased adaptor, it covers all of the Protoboard Jumper-pin Holes (not protoboard friendly).

An Unplanned Back-to-Back Configuration

But, . . . while playing with this Flat-Wing Configuration, I notice that if I reverse the two boards the Esp-12 fits very nicely on top and leaves most of the Protoboard Jumper-pin holes available. I clipped the Header-pins very close (short) with very sharp clippers. And then, just for good measure and avoid shorts, I put tape on the back side of the Esp-12 board.

Adaptors Back-to-Back

The Esp-12 is aligned and then soldered on top.

Esp-12 Soldered to Back-to-Back Adaptors

Esp-12 Read for Use

Or, Ready for 8-Pin DIP Socket

Note, with this Header-pin spacing, this would fit into a normal 8-pin DIP socket !

The Family

Here is my (new) current family of Esp-12, ready for action.

The Family

Let the Esp Programming Fun Begin, . . . Again !! :-)


BTW, I have a Esp-13 Adaptor being manufactured, more on this later.


UPDATE:
I like the Back-to-Back Configuration so well, that I plan to create yet another adaptor, it will be a single board Back-to-Back Adaptor - details to follow.


-- Home Page: https://WA0UWH.blogspot.com

Esp8266-Esp11 Adaptors

UPDATE:
See below

I prefer the Esp8266; Esp11's and Esp12's for my projects, but interfacing to tightly spaced pins can be challenging. In past (see previous blogs) I have used several methods to make the connection to the edge "Castellated" pin-outs.

What is needed is an adaptor for the Esp11's 1.27mm (50mil) pin spacing and the standard 100mil protoboard spacing.

The following is my solution for the Esp11, a small board produced by OSHPark.com. I have made this board available for your use via a shared project, at:

https://oshpark.com/shared_projects/jPGyhNsp

Let me know if you find these useful, Thanks.

The Adaptors are so inexpensive, 3 for $1.00, that I suggest ordering them in groups of 12 or 24, the larger count makes the postage and PayPal charges more tolerable for OshPark.

A group of Adaptors

Ready for Solder

Castellated Connector

The Castellated Connector is created by the manufacture by routing (cutting) though the centre of a plated hole.

Note: the hole conductor is slightly "curled" (distorted) by the cutting process, but that is not a problem as the curl will be encased in solder when used.

Pin Alignment with Esp8266-Esp11

The Pin Pitch is 1.27mm (50mils)

Solder Is Used as Jumpers

The Solder Jumpers are very strong and durable, and perhaps making the joint stronger than the fiberglass PCB itself.

Solder Jumpers were Installed in Groups

Pin Headers (100mil) are Added

Ready for Use

These four Esp's are now ready to be used on my protoboard.

Some of my Esp projects are available online, see previous posts.


-- Home Page: https://WA0UWH.blogspot.com

Esp8266 - My Experimental WebServer

I have been developing an Esp8266 Experimental WebServer (see previous posts).

There are many issues that had to be over come, and still more that will be worked on in the future. At one point I was just about to give-up on this effort, but my friend Jeff - KO7M has helped with some very sticky issues.
 
Currently my WebServer is about 2500 lines of Arduino IDE code, and compiles to about 300K bytes.

The WebServer provides three major web pages; Home, Help and Admin. Some of these pages are "contrived" to be large (e.g., ~30Kb), to push the limits of; Memory, Heap, and WIFI data transfers. Many issues had to be solved to allow this to all work. Heap and Memory Management has always been an issue. For String Constants, I use a P() and Pn() functions throughout (see code). For graphics, I have used SVG.

Here is a list of current features:

 
/*
 * This Program Provides the following for the Esp8266 as a WebServer:
 *      Home Page Dashboard
 *      Help Page
 *      Admin Page, with WiFi Scan and Network Selection
 *      Page Navigator and Resource Links
 *      Raw Data Query
 *      NTP Network Time (Unix Epoch)
 *      mDNS Service/Client
 *      Optional Page AutoRefresh
 *      Auto AP Timeout
 *      Implements SVG 12/24 Hour Clock
 *      Implements SVG Gauge for FreeMem and Vdd
 *      Implements and uses Html "meter" function
 * 
 */

This WebServer is a little slow, but tests indicate that it will support several browser connections, although only one will be actively be transferring data.

I am going to post my work to GitHub for others to use and/or provide feedback, I will provide the Link here when it is setup.

This WebServer is some what "contrived", and which I will use as a Template for my future Esp8266 projects. Normally, an Esp8266 would be used for data collection or control, but I wanted at least a simplified Stand alone Dashboard and with easy data access and I/O control, I think my implementation fits the requirements.

Here are a few of the graphic elements that are implemented, all of which are sourced directly by the Esp8266:

Esp8266 Dashboard
(home page)

An SVG Graphic

Admin Wifi Scan Page

Occasionally, I will make one of my Esp8266 WebServer available online at: dc02.ebcon.com:8160

There are still more functions and ideas that I want to implement. And, there is lots-of-room for optimization, but I want this effort to be as dynamic and free of limitations as possible.

I will continue to develop this and try to keep the GitHub code up-to-date.


-- Home Page: https://WA0UWH.blogspot.com

Esp8266 - Including Two New Functions

I am working on two new functions for my Esp8266 Project Template (see previous post). The two new functions are:

• NTP Client to obtain Unix Time, and Provide Low Quality Time as an NTP Server
• Configuration for New Node via a query to a Previously Configured Node.

The Idea is that a newly constructed Esp8266 node should be able to configure itself based on similar node in the immediate area. The new node will be able to obtain SSID, Passwd and maybe Epoch (Unix Time) from a working node. Once the new node is online, a more accurate time will be obtained from a national NTP server (i.e., pool.ntp.org).

The full list of obtained configuration items have yet to determined.

I have some of this working, but it is still work in progress.


-- Home Page: https://WA0UWH.blogspot.com

Esp8266 - Project Progress

For the last several weeks I have been programming my Esp8266's, I have experienced frustration, anguish, and some delight.

The frustration comes from the fact that the Esp8266 is a relatively new product and not all (or enough) documentation has been translated from Chinese to English. Or, the six or so, development environments (i.e. IDE's) are implementing conflicting strategies for supporting libraries and functions. Sorting out what is relevant, is sometimes difficult

So far, I have been using the Arduino IDE (originally Rev 1.6.1) which support the Esp8266. The included Examples are nice, but are not very useful as for an actual application, that is, they do not suggest how to implement typical combined solutions for some rudimentary functions.

For example, I think a typical home-grown project should have the ability to run in Software Access Point (SoftAP) mode at initial turn-on and accept configuration or connection information for Station (STN) mode connecting to local Access Point (AP) for further operations.

My Goal

My goal is to build that function into my project, but the details lacking on how to switch from one Channel or SSID to another via software control. The Esp8266 can be easily "Programmed" to attach to a selected AP, but that requires a new download for each environment. I am sure the solution is simple, but so far it eludes me.

Still Learning

I have learned a lot, and I have many of my desired functions working.

Note: two things I have learned about the Arduino IDE that I did NOT know, and documented here for my future reference, are;

• First, Arduino IDE Tab Sections (normally files) with a hidden ".ino" suffix are compiled as part of the "main" file ("*.ino") without the need for the usual compiler "forward" declaration and "export" declarations typically found in other software IDE's. This is very handy for quick small projects, where organization is more important than software development correctness.

• Second, Arduino IDE "control /" will convert a section of code into a "comment" and back again. This is very useful while debugging. This short-cut has always been there (I guess) but I had not previously used it, but now I use it all the time.

Project Template

In general, I am creating a "template" for my future projects. The template will have the following functions:

• A "Dashboard" page which is accessible via the assigned device IP Address or alias.
• A "Help" page that will list all of the available functions
• "Query" methods or functions to extract data, that can be used via a web page or non-typical web access methods (e.g., curl, telnet, etc).
• An "Admin" page, to control different aspects of the device.
• A "Navigator Bar" to switch between pages and "Links" to support information.
• A mDNS Responder, so that a device can be access via a "*.local" alias.
• My goal is to include MQTT functions

Here are some example Pages, The Navigator Bar can be used to switch pages. Note: code was included to make the pages, Desktop and Mobile Device Friendly:

Home Page - The Dashboard

This is the "Admin Page" after a WiFi Scan, network selection and Passphrase can be entered to switch networks.

Admin Page - WiFi Scan

 The Help Page provides information and links.

Help Page

Note: the Help page provides links and/or URL's so that a "Query" can easily be constructed for Raw Data access. For example, on a remote system:

$ curl Nod1590.local/q/uptime 

The above will prints just the number of seconds (uptime) since power-on.

Some Hurtles

One of the most troublesome aspects of my Esp8266 development was dealing with unexpected web request. Initially, my tests were simple and successful, I was using a Firefox browser on my LinuxMint Workstation, where a simple single web requests received a simple single response from the Esp8266 App. The app provides a "404 Error" for all unexpected web request. This all worked great, and as expected.

But Then . . .

But then, I tried my app using my Cell Phone and Tablet Browsers, which quickly crashed my Esp8266 Web Application. The problem was these browsers repeatedly (and relentlessly) request a "/favicon.ico" file, which over-ran the Esp8266 buffers, or just "hogged" all of the available bandwidth ( which I have very little, I have a slow and terribly BAD Internet connection to my Shop ). The "/favicon.icon" requests can NOT be "ignored", they just keep coming and therefore the Esp8266 must deal with them. I spent several days (actually more than a few) looking for a solution, including creating and sending a dummy ( or contrived ) files to satisfy the requests. The online searches provided many suggestions, but in all cases the solutions were not optimal, they all still uses my (scarce) network bandwidth and small Esp8266 buffer space.

My devised and final solution was much more simple than I expected (and elegant, if I may say so). I pre-emptively re-direct all external "/favicon.ico" requests to where they belong - that is, to Espressif (the Manufacture) web site where they happily (or at least hopefully so) provide the file. The redirection takes place in the Browser and therefore the data transfer is between the browser and manufacture, and NOT with my Esp application. Espressif gets the web hits and I get bandwidth relief, the user sees a nice icon on the Exp8266 App browser tab, and my application does not have to deal with the issue. A single line in the Esp's web application html Header does the trick. Very-Very Simple!

<link rel="shortcut icon" href="http://espressif.com/favicon.ico"></link>

Note: Yes, I know this solution may not be web polite, and if I find a better solution, I will post it.

Lack of Understanding

Another thing that plagued my early Esp8266 App development was my lack of understanding of the "Strings Objects"! I have been working with too many languages lately. My friend Jeff - KO7M has helped and provided suggestion. In general, I was originally inappropriately using Strings that could be trashed (and crashed) by the "Garbage Collection" (GC) process. Most web page generation is about generating Large strings, managing them is important so that the "Heap" is not stressed and so GC can properly reclaim space.

But Now, Progress

Also, to save RAM space, I am using a previously used technique (see previous post) of moving all "string constants" to FLASH using PROGMEM, and "not" using the Heap, and there by avoiding some of the RAM space issues. As stated above, most web pages generation is just manipulating long strings in RAM.

To Be Done

I just have a few more issues to work out, and then I will publish the code as an example of Stand-alone Esp8266 Development App.


-- Home Page: https://WA0UWH.blogspot.com

Esp8266 Esp-12

There are several configurations of the Esp8266 modules which is denoted with a Exp-XX suffix. Currently I know of Esp-01 through Esp-12, each have their advantages and dis-advantages.

Although I have be experimenting with the Esp8266 Esp-11 (see previous post), my goal is to use the Esp-12 configuration for more complex applications. The Esp-11 configuration has only 8 pins, while the Esp-12 has 16 pins, and therefore more I/O options.

Breakout boards are available for each, but there are trade-offs; the Esp-12 Breakout Board is much wider than the Esp-12 itself. Even though the breakout Esp-12 Breakout Board is protoboard friendly (i.e., 0.1in spacing), it is so wide that there is little space to attach jumpers (I think only one hole row on each side is usable).

I decided to try the technique of bending header pins as necessary to match the 2mm spacing, I originally used this method on the Esp-11 (see previous post).

The pin spacing of the Exp-11 is smaller at 1.27mm, while the Esp-12 is 2.0mm. A good microscope is needed to solder headers directly onto either.

Esp8266 Esp-12 with two 0.1in Headers Attached

Now all that is needed is wiring-it-up and some software (Yeah, . . . famous last words).

-- Home Page: https://WA0UWH.blogspot.com