Merge lp://qastaging/~epics-core/epics-base/rsrvbindiface into lp://qastaging/~epics-core/epics-base/3.15

Rebased against 3.15

When building for VxWorks 5.5.2 (which we still support):

../udpiiu.cpp: In method `udpiiu::udpiiu(epicsGuard<epicsMutex> &, epicsTimerQueueActive &, epicsMutex &, epicsMutex &, cacContextNotify &, cac &, unsigned int, tsDLList<SearchDest> &)':
../udpiiu.cpp:169: passing `int *' as argument 4 of `setsockopt(int, int, int, char *, int)'

The fix is to cast &flag to (char *) [I tried (void *), but the C++ rules don't allow that].
Later, same target-arch:

../../../src/ioc/rsrv/caservertask.c: In function `tryBind':
../../../src/ioc/rsrv/caservertask.c:135: warning: passing arg 2 of `bind' from incompatible pointer type
../../../src/ioc/rsrv/caservertask.c: In function `rsrv_grap_tcp':
../../../src/ioc/rsrv/caservertask.c:201: parse error before `alen'
../../../src/ioc/rsrv/caservertask.c:202: `alen' undeclared (first use in this function)
../../../src/ioc/rsrv/caservertask.c:202: (Each undeclared identifier is reported only once
../../../src/ioc/rsrv/caservertask.c:202: for each function it appears in.)
../../../src/ioc/rsrv/caservertask.c: In function `rsrv_init':
../../../src/ioc/rsrv/caservertask.c:579: warning: passing arg 4 of `setsockopt' from incompatible pointer type

On line 135 change bind() arg #2 to be (struct sockaddr *) &addr->sa (cast is needed to drop const).
Line 201 is not standard C code, a declaration appears after other statements in block; swapped with preceding line.
Line 579 needs another cast to (char *).

The result builds on all my standard architectures.

Question: Is 'rsrv_grap_tcp' a typo for 'rsrv_grab_tcp'?

Could your point about there being a broadcast route through lo conceivably apply to other OSs too? If so I would prefer that we use some other conditional in the libCom/osi/os/default/osdNetInf.c file to enable that code; adding a port to a new OS should be possible without having to make *any* changes to the existing code. I would define some macro in osi/os/Linux/osdSock.h and use that instead of __linux__ in osdNetInf.c.

More to come, I'll try to do some testing this afternoon.

On 01/20/2016 02:10 PM, Andrew Johnson wrote:
> Question: Is 'rsrv_grap_tcp' a typo for 'rsrv_grab_tcp'?

I'm going to go with, yes.

On 01/20/2016 02:10 PM, Andrew Johnson wrote:
> Could your point about there being a broadcast route through lo conceivably apply to other OSs too?

I suspect so as it seems like a reasonable thing to do. However, I
couldn't find any reference, and am not in a position to test any other
targets except RTEMS (where it isn't useful).

> ... I would define some macro in osi/os/Linux/osdSock.h and use that instead of __linux__ in osdNetInf.c.

Seems reasonable.

I'm not quite sure what this means, but on my Windows & Cygwin box:

ECLIPSE$ route PRINT -4 127*
===========================================================================
Interface List
 12...ac 16 2d 0d 54 de ......Intel(R) 82579LM Gigabit Network Connection
  1...........................Software Loopback Interface 1
 13...00 00 00 00 00 00 00 e0 Microsoft ISATAP Adapter
 11...00 00 00 00 00 00 00 e0 Microsoft 6to4 Adapter
===========================================================================

IPv4 Route Table
===========================================================================
Active Routes:
Network Destination Netmask Gateway Interface Metric
        127.0.0.0 255.0.0.0 On-link 127.0.0.1 306
        127.0.0.1 255.255.255.255 On-link 127.0.0.1 306
  127.255.255.255 255.255.255.255 On-link 127.0.0.1 306
===========================================================================
Persistent Routes:
  None

It's likely that Cygwin would see those same routes, and it uses the osi/os/default implementation (how would I test that?). We might want to add similar code to the osi/WIN32/osdNetIntf.c but I don't really understand what that's doing at line 134, so I'm not going to suggest we make any changes to that right now.

Seems to be working for me on VxWorks 6.9, I can connect to PVs, and outgoing CA client connections still work OK too (although I didn't test setting CA_ADDR_LIST).

Also works with a second network interface configured at 10.0.0.2 (but not actually connected to anything):

iocanj> casr 2
Channel Access Server V4.13
No clients connected.
Server interface 0.0.0.0:5064
 UDP receiver 1 0.0.0.0:5064
Beacon destination 10.0.0.255:5065
Beacon destination 164.54.11.255:5065
There are currently 1288 bytes on the server's free list
7 client(s), 0 channel(s), 0 event(s) (monitors) 0 putNotify(s)
0 small buffers (16384 bytes ea), and 0 jumbo buffers (16408 bytes ea)
The server's resource id conversion table:
Bucket entries in use = 0 bytes in use = 16404
Bucket entries/hash id - mean = 0.000000 std dev = 0.000000 max = 0
The server's array size limit is 16408 bytes max
Channel Access Address List
10.0.0.255:5065
164.54.11.255:5065
value = 19 = 0x13

The "Server interface 0.0.0.0:5064" and "UDP receiver 1 0.0.0.0:5064" lines are slightly confusing in the above status message, although I guess they do make sense. At least the casr command now shows the dynamically allocated TCP port number (which used to only show up at iocInit, making it hard to look up later on).

when running on an IOC without DNS properly configured the two entries in the CA Address list at the bottom take a minute or two each to print while waiting for the DNS search to time out; in my experience network administrators don't add names for their broadcast IP addresses to DNS, so why are we querying DNS before printing just the numbers in that list? I guess there might be host IPs in there too, but in this case I would be happy with just printing the numbers even for them. You probably haven't touched this part of the code though...

Pointing EPICS_CAS_INTF_ADDR_LIST to my secondary interface's IP address stopped CA searches from the primary network being seen/replied to.

When I pointed EPICS_CAS_INTF_ADDR_LIST to my primary interface's broadcast address though, the error message was not quite as helpful as it could be:

Starting iocInit
############################################################################
## EPICS R3.15.3-DEV $$Date: Sun 2015-11-22 18:10:40 +0100 $$
## EPICS Base built Jan 20 2016
############################################################################
Warning: RSRV has empty beacon address list
CAS: Socket bind 164.54.11.255:5064 error was "S_errno_EINVAL"
Thread tShell0 (0x6f6c70) can't proceed, suspending.

Users are going to try putting broadcast addresses in EPICS_CAS_INTF_ADDR_LIST (since they're used to using them on the client side), so if we aren't going to recognize and fix that setting automatically, there should be some kind of diagnostic explaining what the problem is in a bit more detail than is shown above. Calling cantProceeed() in that case is probably correct though.

How might I go about testing the multi-cast support?

This branch at least needs an entry in the Release Notes explaining what's new and how to configure the new features. I'm not sure if there's a suitable place in the AppDevGuide for more detail or not, although I think we...

So this branch was overtaking my rebase on GitHub... need to rearrange.

I pushed the fix for the issue with compiling on MinGW to the 3.15 branch (http://bazaar.launchpad.net/~epics-core/epics-base/3.15/revision/12717), so that should be handled.

Do we need more detailed test instructions?
Remember that we wanted to trigger a serious testing campaign for this, so we should have some kind of rough test plan that we want people to run.

> Do we need more detailed test instructions?

We do, so I'll write something up.

Committed fixes for building on VxWorks 5.5.2.

Fixed incorrect broadcast address list construction. Made the "no addresses" error message longer. Added release notes. Teach PCAS to listen for mcast search requests as well.

> ... the two entries in the CA Address list ...

This isn't actually the CA address list, but rather the beacon address list printed a second time. The first uses ipAddrToDottedIP(), and the second is now removed.

FYI the only place I've found the actual CA address list (for the dbCa context) is buried in "dbcar '' 8".

> ... How might I go about testing the multi-cast support? ...

Set:

EPICS_CAS_INTF_ADDR_LIST='0.0.0.0 224.0.2.9' EPICS_CA_ADDR_LIST='224.0.2.9' EPICS_CA_AUTO_ADDR_LIST=NO

Since this branch allows 127.255.255.255 to be discovered and included in the default address list (for Linux), some configurations will result in more spurious duplicate PV warnings when IOC and client are on the same system. I go back and forth about whether this is acceptable or not. Thoughts?

You broke the generation of beacon anomalies at IOC startup somehow.

There is a tool build as part of src/ca called casw which monitors for and displays beacon anomalies, using exactly the same code that the clients use. When I start a regular 3.14 softIoc with casw running I get the expected anomalies displayed, but when starting a softIoc from this branch it shows nothing.

The IOC is sending out beacons at the usual exponentially-decaying rate since I do see them displayed when I ask for every beacon by running 'casw -i 2':

  localhost:5064 2016-01-21 12:56:35.851115672
  tux.aps.anl.gov:5064 2016-01-21 12:56:35.851214480
  localhost:5064 2016-01-21 12:56:35.871209057
  tux.aps.anl.gov:5064 2016-01-21 12:56:35.871263061
  localhost:5064 2016-01-21 12:56:35.911352969
  tux.aps.anl.gov:5064 2016-01-21 12:56:35.911395578
  localhost:5064 2016-01-21 12:56:35.991482300
  tux.aps.anl.gov:5064 2016-01-21 12:56:35.991524021
  localhost:5064 2016-01-21 12:56:36.151682412
  tux.aps.anl.gov:5064 2016-01-21 12:56:36.151731490
  localhost:5064 2016-01-21 12:56:36.471809095
  tux.aps.anl.gov:5064 2016-01-21 12:56:36.471859590
  localhost:5064 2016-01-21 12:56:37.111908997
  tux.aps.anl.gov:5064 2016-01-21 12:56:37.111952211
  localhost:5064 2016-01-21 12:56:38.392053916
  tux.aps.anl.gov:5064 2016-01-21 12:56:38.392075091
  localhost:5064 2016-01-21 12:56:40.952238006
  tux.aps.anl.gov:5064 2016-01-21 12:56:40.952306370
  localhost:5064 2016-01-21 12:56:46.072448531
  tux.aps.anl.gov:5064 2016-01-21 12:56:46.072515535

I tried starting the IOC with EPICS_CAS_AUTO_BEACON_ADDR_LIST=NO EPICS_CAS_BEACON_ADDR_LIST=tux and while the output from 'casr 2' shows only one beacon destination address, I still get no joy with the anomaly detection from casw.

> You broke the generation of beacon anomalies at IOC startup somehow.

Good catch, and now fixed. I had checked that beacons were sent, but didn't look at the contents. The counter wasn't being copied into the packet buffer, so it was always sending a count of 0.

Another (intentional) change is that beacons are sent with the "address" field set to zero, so carepeater will fill it in based on the source address. With the switch to an unconnected socket this gives the correct result (verified with casw and wireshark).

Beacons fixed, thanks.

Trying multicast name resolution:

tux$ EPICS_CAS_INTF_ADDR_LIST='0.0.0.0 224.0.2.9' EPICS_CA_ADDR_LIST=224.0.2.9 EPICS_CA_AUTO_ADDR_LIST=NO bin/linux-x86_64/softIoc -x anj
Starting iocInit
############################################################################
## EPICS R3.15.3-DEV $$Date: Sun 2015-11-22 18:10:40 +0100 $$
## EPICS Base built Jan 21 2016
############################################################################
CAS address list can not contain 0.0.0.0 and other addresses, ignoring...
iocRun: All initialization complete
epics>

I get the same output from 'casr 2' without the warning message by omitting the 0.0.0.0 interface address, so I'll assume including it was just a mistake in your instructions above.

Searches from the same subnet succeed with EPICS_CA_ADDR_LIST=224.0.2.9 EPICS_CA_AUTO_ADDR_LIST=NO. Do I need to have our network admins configure something for searches to work across subnets? They don't for me at the moment using the same unmodified 3.14-based client. This was only an experimental feature though so it's not a blocker at all.

If I configure the IOC to only use the localhost interface, why does it still send beacons through my primary interface?

tux$ EPICS_CAS_INTF_ADDR_LIST=localhost bin/linux-x86_64/softIoc -x anj
Starting iocInit
############################################################################
## EPICS R3.15.3-DEV $$Date: Sun 2015-11-22 18:10:40 +0100 $$
## EPICS Base built Jan 21 2016
############################################################################
iocRun: All initialization complete
epics> casr 2
Channel Access Server V4.13
No clients connected.
Server interface 127.0.0.1:5064
 UDP receiver 1 127.0.0.1:5064
 UDP receiver 2 127.255.255.255:5064
Beacon destination 127.255.255.255:5065
Beacon destination 164.54.11.255:5065
...

I had expected that explicitly setting the interface would also configure the beacons to be sent through the same interface(s), although I do realize the original code didn't work like that and it can be done manually. I'd be OK with leaving that change for later if there are objections to fixing it here though.

I also have some minor modifications to casr which I'll finish off next week.

> Trying multicast name resolution:
>
> tux$ EPICS_CAS_INTF_ADDR_LIST='0.0.0.0 224.0.2.9' EPICS_CA_ADDR_LIST=224.0.2.9
> EPICS_CA_AUTO_ADDR_LIST=NO bin/linux-x86_64/softIoc -x anj
> Starting iocInit
> ############################################################################
> ## EPICS R3.15.3-DEV $$Date: Sun 2015-11-22 18:10:40 +0100 $$
> ## EPICS Base built Jan 21 2016
> ############################################################################
> CAS address list can not contain 0.0.0.0 and other addresses, ignoring...
> iocRun: All initialization complete
> epics>
>
> I get the same output from 'casr 2' without the warning message by omitting
> the 0.0.0.0 interface address, so I'll assume including it was just a mistake
> in your instructions above.
>
> Searches from the same subnet succeed with EPICS_CA_ADDR_LIST=224.0.2.9
> EPICS_CA_AUTO_ADDR_LIST=NO.

Hm.
Shouldn't there be a way to specify: "bind to all available network interfaces, plus the extra multicast address".

It look s like the settings from Michael's instructions were trying to do this, but the "0.0.0.0" was rejected. (Which might be wrong.)
Do searches from the same subnet with broadcasts or directed to the IOCs IP work in this configuration?

Ha, I should be more careful about testing exactly the examples I give.

> EPICS_CAS_INTF_ADDR_LIST='0.0.0.0 224.0.2.9'

Triggered the error.

> EPICS_CAS_INTF_ADDR_LIST='224.0.2.9 0.0.0.0'

Didn't.

Now fixed so that ordering shouldn't matter.

Also, EPICS_CAS_INTF_ADDR_LIST='224.0.2.9' should be equivalent to both of these. If the interface list is empty after mcast destinations are removed, then the wildcard is added.

To actually trigger this error (now cantProceed) include the wildcard address and a specific interface address.

> Do I need to have our network admins configure something for searches to work across subnets?

I'm fairly certain that I'm doing the right thing on the socket level. But I just don't know how what additional work is needed at the router/switch level. As I think about it, this is probably a good tech-talk question.

For layer 2 devices (hubs and dumb switches) multicasts and broadcasts are quite similar. In the presence of layer 3 (routers and "smart" switches) things will be different. I'm not sure how to test this situation. Note that how this works, and what configuration is needed, may well depend on which mcast address is used.

More changes to beacon address list population (I think I've got it this time).

https://wiki-ext.aps.anl.gov/epics/index.php/RSRV_Configuration_Test_Plan

Details most tested configurations with expected results.

I have a set of changes which significantly tidy up the output from casr. Should I commit them here, or to 3.16 after this branch has been merged up?

Here's some new output:

epics> casr
Channel Access Server V4.13
2 client(s) connected.

epics> casr 1
Channel Access Server V4.13
2 client(s) connected:
    CA-client at 164.54.9.24:57238 'tux.aps.anl.gov':
 User 'anj', V4.13, Priority = 0, 2 Channel(s)
    CA-client at 164.54.8.21:33434 'venus':
 User 'anj', V4.11, Priority = 0, 1 Channel(s)
CAS-TCP server 0.0.0.0:5064
    CAS-UDP name server 0.0.0.0:5064
2 CAS-beacon destinations:
    127.255.255.255:5065
    164.54.11.255:5065

epics> casr 2
Channel Access Server V4.13
2 client(s) connected:
    CA-client at 164.54.9.24:57238 'tux.aps.anl.gov':
 User 'anj', V4.13, Priority = 0, 2 Channel(s)
        Channel: 'anj:exit'
        Channel: 'anj:exit.VAL[0]{"ts":{}}'
    CA-client at 164.54.8.21:33434 'venus':
 User 'anj', V4.11, Priority = 0, 1 Channel(s)
        Channel: 'anj:exit.A'
CAS-TCP server 0.0.0.0:5064
    CAS-UDP name server 0.0.0.0:5064
2 CAS-beacon destinations:
    127.255.255.255:5065
    164.54.11.255:5065

epics> casr 3
Channel Access Server V4.13
2 client(s) connected:
    CA-client at 164.54.9.24:57238 'tux.aps.anl.gov':
 User 'anj', V4.13, Priority = 0, 2 Channel(s)
        Channel: 'anj:exit'
            field_type=DBF_DOUBLE (8 bytes), dbr_type=DBF_DOUBLE, 1 element, no filters
            # on eventq=1, access=rw
        Channel: 'anj:exit.VAL[0]{"ts":{}}'
            field_type=DBF_DOUBLE (8 bytes), dbr_type=DBF_DOUBLE, 1 element
            2 filters (1 pre eventq, 0 post eventq)
            final field_type=DBF_DOUBLE (8B), 1 element
            # on eventq=1, access=rw
    CA-client at 164.54.8.21:33434 'venus':
 User 'anj', V4.11, Priority = 0, 1 Channel(s)
        Channel: 'anj:exit.A'
            field_type=DBF_DOUBLE (8 bytes), dbr_type=DBF_DOUBLE, 1 element, no filters
            # on eventq=1, access=rw
CAS-TCP server 0.0.0.0:5064
    CAS-UDP name server 0.0.0.0:5064
2 CAS-beacon destinations:
    127.255.255.255:5065
    164.54.11.255:5065

epics> casr 4
Channel Access Server V4.13
2 client(s) connected:
    CA-client at 164.54.9.24:57238 'tux.aps.anl.gov':
 User 'anj', V4.13, Priority = 0, 2 Channel(s)
 Task Id = 0x7f1e2c005680, Socket FD = 7
 10.30 secs since last send, 10.30 secs since last receive
 Unprocessed request bytes = 0, Undelivered response bytes = 0
 State = up
        Channel: 'anj:exit'
            field_type=DBF_DOUBLE (8 bytes), dbr_type=DBF_DOUBLE, 1 element, no filters
            # on eventq=1, access=rw
        Channel: 'anj:exit.VAL[0]{"ts":{}}'
            field_type=DBF_DOUBLE (8 bytes), dbr_type=DBF_DOUBLE, 1 element
            2 filters (1 pre eventq, 0 post eventq)
                Array (arr): start=0, incr=1, end=0
                Timestamp (ts)
            final field_type=DBF_DOUBLE (8B), 1 element
            # on eventq=1, access=rw
    CA-client at 164.54.8.21:33434 'venus':
 User 'anj', V4.11, Priority = 0, 1 Channel(s)
 Task Id = 0x7f1e2c005f80, Socket FD = 9
 11.35 secs since last send, 11.35 secs since last receive
 Unprocessed request bytes = 0, Undelivered response bytes...

No strong preference.

But since the changes are probably in one source file and thus easy to separate from the functional changes, why not here.

Well I think the answer is simple and none technical. Commit it if you have time to update the wiki ;)

Ok; the changes are mostly in rsrv/caservertask.c, but I also added more interest-level conditionals to dbChannelShow() and modified printf() strings in all the std/filters channel_report() methods and in libCom's bucketShow(). I will commit and make the changes on the Wiki page to match.

I did notice that casr() no longer prints a log_one_client() report for the UDP (i.e. name-server) clients. There used to be only one such client thread, but now the number can vary, with one or two threads for each interface. I had removed the UDP reporting code from the log_one_client() routine since it was no longer being used, but I just put it back in case we want to add these threads back into the report. I think I know how to do that if it's wanted.

How are the multicast monitors supposed to work? There isn't a separate thread started for them, the code just joins every primary UDP socket to the multicast group(s) configured. Isn't that going to cause multiple search replies from any IOC that has two interfaces say if both receive a copy of the same multicast search request? We might need a way to configure which multicast addresses get joined to which interfaces if this starts to be used much.

I added 2 comments to the wiki page in bold red, the beacon list for Test #2 seems wrong, and Test #6 gives me different results (the interface on 0.0.0.0 is still open, so clients in the same subnet can still connect without having to use the multicast address).

I think I'm done with this now; apart from multi-cast it builds and works fine.

> How are the multicast monitors supposed to work? ... Isn't that going to cause multiple search replies from any IOC that has two interfaces say if both receive a copy of the same multicast search request?

The best I can say is probably not, at least on Linux. I haven't found any clear description, of all that is involved here, so I could be completely wrong. As I understand it, as with any ip packet, a multicast packet is always received on a specific interface, and won't be forwarded absent some specific routing rule ("ip mroute" is empty by default). So an mcast should only be delivered to sockets bound to the interface it was received on, and joined to the group.

Linux might have some additional hoops to jump. I find some references to needing to disable reverse path filtering (rp_filter), though this isn't enabled by default w/ debian 8.

Once the test-plan wiki page is deemed correct, and the code to match, we had discussed putting out a request for testing on tech-talk. I'd like to have this tested in a more realistic setting.

> the beacon list for Test #2 seems wrong

Not surprising. For some reason I'm getting confused about whether this should match the server or client interface address list. I've "fixed" this at least twice now and two parity flips means I need some outside input. Can you fix the wiki page to what you think it should be?

A given socket will only send multicast packets out a single interface (as determined by the local routing table or IP_MULTICAST_IF). So no duplication there.

There probably should be some way of configuring IP_MULTICAST_TTL though. Would you care to pick a EPICS_CA_* name for this Andrew?

> Test #6 gives me different results ...

... Due to the way my test system is configured :P You result is the correct one.

> ... We might need a way to configure which multicast addresses get joined to which interfaces if this starts to be used much.

> ... as determined by the local routing table or IP_MULTICAST_IF ...

Yes, to allow full flexibility the both client and server address lists would need to be configured with pairs of multicast+interface address. However, I'm inclined to forgo this unnecessary until someone describes a situation where this is necessary, or even helpful.

Beacons should be related to the server interfaces in use. In the past they weren't, originally because there were no EPICS_CAS_ variables, although later I think Jeff just re-used the contents of EPICS_CA_ADDR_LIST for rsrv's beacon list unless you explicitly set EPICS_CAS_BEACON_ADDR_LIST. I don't think the code is quite right yet, although it seems fairly close.

Starting a softIoc with all defaults, I get this:
tux$ bin/linux-x86_64/softIoc -x anj
...
CAS-TCP server on 0.0.0.0:5064 with
    CAS-UDP name server on 0.0.0.0:5064
Sending CAS-beacons to 2 addresses:
    127.255.255.255:5065
    164.54.11.255:5065

If I set EPICS_CAS_INTF_ADDR_LIST to my primary IP address, I get this:
tux$ EPICS_CAS_INTF_ADDR_LIST='tux' bin/linux-x86_64/softIoc -x anj
...
CAS-TCP server on 164.54.9.24:5064 with
    CAS-UDP unicast name server on 164.54.9.24:5064
    CAS-UDP broadcast name server on 164.54.11.255:5064
Sending CAS-beacons to 1 address:
    164.54.11.255:5065

Setting it to localhost I get this:
tux$ EPICS_CAS_INTF_ADDR_LIST='localhost' bin/linux-x86_64/softIoc -x anj
...
CAS-TCP server on 127.0.0.1:5064 with
    CAS-UDP unicast name server on 127.0.0.1:5064
    CAS-UDP broadcast name server on 127.255.255.255:5064
Sending CAS-beacons to 1 address:
    127.255.255.255:5065

I can add addresses to the beacon list:
tux$ EPICS_CAS_INTF_ADDR_LIST='localhost' EPICS_CAS_BEACON_ADDR_LIST=224.0.2.9 bin/linux-x86_64/softIoc -x anj
...
CAS-TCP server on 127.0.0.1:5064 with
    CAS-UDP unicast name server on 127.0.0.1:5064
    CAS-UDP broadcast name server on 127.255.255.255:5064
Sending CAS-beacons to 2 addresses:
    127.255.255.255:5065
    224.0.2.9:5065

Those all look correct to me so far, beacons are being sent to the broadcast addresses of the interfaces that my servers are listening on, plus any others I explicitly specify. However if I set EPICS_CA_AUTO_ADDR_LIST=NO the interface list gets ignored for beacons and I get the 'empty beacon address list' warning:
tux$ EPICS_CAS_INTF_ADDR_LIST='localhost' EPICS_CA_AUTO_ADDR_LIST=NO bin/linux-x86_64/softIoc -x anj
...
Warning: RSRV has empty beacon address list
...
CAS-TCP server on 127.0.0.1:5064 with
    CAS-UDP unicast name server on 127.0.0.1:5064
    CAS-UDP broadcast name server on 127.255.255.255:5064
Sending CAS-beacons to 0 addresses:

In this case it still seems to be copying the initial beacon list from the EPICS_CA_ADDR_LIST instead of the interface list because I can put addresses in the beacon list by adding them to EPICS_CA_ADDR_LIST, e.g:
tux$ EPICS_CAS_INTF_ADDR_LIST='localhost' EPICS_CA_ADDR_LIST=224.0.2.9 EPICS_CA_AUTO_ADDR_LIST=NO \
bin/linux-x86_64/softIoc -x anj
...
CAS-TCP server on 127.0.0.1:5064 with
    CAS-UDP unicast name server on 127.0.0.1:5064
    CAS-UDP broadcast name server on 127.255.255.255:5064
Sending CAS-beacons to 1 address:
    224.0.2.9:5065

A different issue: If I put a multicast address in EPICS_CAS_BEACON_ADDR_LIST without explicitly setting EPICS_CAS_INTF_ADDR_LIST, casr reports that the server is monitoring that address, which doesn't seem right. It didn't do that when I put it in EPICS_CA_ADDR_LIST (above):
tux$ EPICS_CAS_BEACON_ADDR_LIST=224.0.2.9 bin/l...

> Beacons should be related to the server interfaces in use.

Ok, auto beacon list is no longer tied to client auto address list.

Also, the server interface list will no longer use the beacon address list as a fallback.

Thanks, I think this makes much more sense. The behavior change does need documenting though, in the Release Notes entry and probably in CAref.html as well. I can update the Wiki page to match if you like.

> The behavior change does need documenting though, in the Release Notes entry and probably in CAref.html

Which raises the question: what about PCAS?

> I can update the Wiki page to match if you like.

Please do. And you may assume in future that I would always like.

> Beacons should be related to the server interfaces in use.

I made the corresponding change to PCAS as well.

I also made an attempt to teach RSRV about EPICS_CAS_IGNORE_ADDR_LIST, which I think matches PCAS wrt. configuration options.

Updated CAref.html and release notes.

Unfortunately I think your PCAS changes have broken something. I can run the base-3.15 excas example with no arguments and I get no errors, but if I run it from this branch I get the following errors, printed at the exponentially-slowing beacon generation intervals:

../../../../../../src/ca/legacy/pcas/io/bsdSocket/casDGIntfIO.cc: CA beacon routing (connect to "164.54.11.255:5065") error was "Invalid argument"
../../../../../../src/ca/legacy/pcas/io/bsdSocket/casDGIntfIO.cc: CA beacon routing (connect to "164.54.11.255:5065") error was "Invalid argument"
../../../../../../src/ca/legacy/pcas/io/bsdSocket/casDGIntfIO.cc: CA beacon routing (connect to "164.54.11.255:5065") error was "Invalid argument"

The IP address in that error message is the broadcast address of my local subnet. Connections to the excas PVs from a local client do succeed, so I think it is listening on the primary interface properly. Running casw while starting the server shows beacons coming from localhost but not from the machine's regular IP address (the opposite of running the base-3.15 version of excas).

I updated the Wiki page to remove my last red question and fix the expected casr output there, and fixed the CAS-beacon list in Test #6.

About the wiki page, I don't think we really need to set EPICS_CA_ADDR_LIST or AUTO_ADDR_LIST when starting the IOC in most of these tests, dbCa will always connect to local channels through the internal channel provider anyway, it doesn't use the network at all. Test #5 and maybe #6 could keep those settings though.

Ok, I think this is fixed as well. PCAS was sending beacons from a connect()d UDP socket as RSRV was. Now changed to use sendto() and leave the origin interface field zero'd, as RSRV now does.

CAS beacon issue fixed, will merge after Ralph's approval.

On Windows, I am not getting any of the thread's messages, type

CAS-TCP server on 10.5.1.1:5064 with
    CAS-UDP unicast name server on 10.5.1.1:5064
    CAS-UDP broadcast name server on 10.5.1.255:5064

Have these messages been removed again?

More testing:

cygwin does show the messages.
MSVC 12.0 (both 64 and 32 bit) and MinGW32 do not.

Rough test for the PCAS changes

On 64bit Linux:
The CA Gateway (trunk) compiles, the (very few) unit tests succeed.
However, beacons and reconnection behavior are not tested by these tests yet.

Ah yes, the code for casr() does use ifdef _WIN32 and doesn't call log_one_client() from that section of code. I just committed a fix.

Sorry if I wasn't clear enough.

The Windows builds show *no* thread related messages, none at all.

[...]
iocRun: All initialization complete
epics> casr 1
Channel Access Server V4.13
No clients connected.
Sending CAS-beacons to 2 addresses:
    172.21.63.255:5065
    192.168.56.255:5065
epics> casr 2
Channel Access Server V4.13
No clients connected.
Sending CAS-beacons to 2 addresses:
    172.21.63.255:5065
    192.168.56.255:5065
epics> casr 3
Channel Access Server V4.13
No clients connected.
Sending CAS-beacons to 2 addresses:
    172.21.63.255:5065
    192.168.56.255:5065
epics>

Bummer: the server list was empty (not being populated) on Windows.
I just committed a fix.

I did a complete test on my windows box, report is as follows.

The cygwin target needs fixing. I'm not sure what's happening there.

Please also add tests on the behavior when another IOC is already running.
It is not covered by the test plan, but I have seen IOCs hanging or exiting.

These tests are pretty tedious. Excellent candidate for automation...

---- snip ----
Test Report rsrvbindiface on Windows 7

Test Plan https://wiki-ext.aps.anl.gov/epics/index.php/RSRV_Configuration_Test_Plan
Version 2016-02-16T14:44

Tested 2016-02-25T09:00 by Ralph
                duration 2 hrs
Version lp:~epics-core/epics-base/rsrvbindiface revision 12762
                    plus commit 12716..12717 from lp:~epics-core/epics-base/3.15
Targets MSVC 12.0 (amd64), MSVC (x86), Cygwin (x86_64), MinGW32 (x86)
                dynamic builds
Machine Latitude E7240 (intel i3) running Windows 7

Test #1 Defaults
                            MSVC 64 MSVC 32 Cygwin MinGW32
    client get pass pass pass pass
    epicsParamShow pass pass pass pass
    casr 1 pass pass pass pass
    dbcar "" 8 pass pass pass pass

Note on Cygwin:
- warning at boot time
    Warning: Duplicate EPICS CA Address list entry "169.254.255.255:5065" discarded
- casr and dbcar (UDP search destination) show this interface address (link-local)
  in addition to the two regular interfaces

Test #2 Client address list only
                            MSVC 64 MSVC 32 Cygwin MinGW32
    client get pass pass pass pass
    epicsParamShow pass pass pass pass
    casr 1 pass pass pass pass
    dbcar "" 8 pass pass pass pass

Note on Cygwin:
- warning at boot time
    Warning: Duplicate EPICS CA Address list entry "169.254.255.255:5065" discarded
- casr shows this interface address in addition to the two regular interfaces

Test #3 Client and Server on a single interface
                            MSVC 64 MSVC 32 Cygwin MinGW32
    client get pass pass FAIL pass
    epicsParamShow pass pass FAIL pass
    casr 1 pass pass FAIL pass
    dbcar "" 8 pass pass FAIL pass

Failure on Cygwin:
- error at boot time
    CAS: UDP Socket bcast bind error: "Cannot assign requested address"
  IOC hangs after that message

Test #4 Client and Server on multiple interfaces
                            MSVC 64 MSVC 32 Cygwin MinGW32
    client get pass pass FAIL pass
    epicsParamShow pass pass FAIL pass
    casr 1 pass pass FAIL pass
    dbcar "" 8 pass pass FAIL pass

Failure on Cygwin:
- error at boot time
    CAS: UDP Socket bcast bind error: "Cannot assign requested address"
  IOC hangs ...

> Please also add tests on the behavior when another IOC is already running.

Good point. Provisionally I think the instruction would be to repeat all tests with a second IOC running w/ default config. Just fixed an mistake here were I used 'errno' instead of 'SOCKERRNO'.

> CAS: UDP Socket bcast bind error: "Cannot assign requested address"

This is a failure to bind a UDP socket to an interface broadcast address. So it seems likely that cygwin is (unavoidably) letting the winsock behavior show through. In caservertask.c, I'vee replaced the three '!defined(_WIN32)' with '!(defined(_WIN32) || defined(__CYGWIN32__))'.

> IPv4 Multicast name lookup not supported by this target

The direct cause is that the macro IP_ADD_MEMBERSHIP is not defined in caservertask.c, which is the test I picked for OS support of multicast. I've added an include of "ws2tcpip.h" to WIN32/osdSock.h, which I think is reasonable.

> These tests are pretty tedious. Excellent candidate for automation...

Agreed, just not sure how given the requirement for superuser, and the OS differences in network configuration. How can the effect of linux tap interfaces be achieved on windows/macos? What sort of virtual NIC should be used?

Tuntap is available as an add-on for OS X — I use it with a Verilog simulator to test the ethernet-in-firmware code produced here by Larry Doolittle.

> On Feb 25, 2016, at 8:42 AM, mdavidsaver <email address hidden> wrote:
>
> How can the effect of linux tap interfaces be achieved on windows/macos?

--
Eric Norum
<email address hidden>

I was doing manual testing here using a combination of localhost and my workstation's primary IP address (i.e. `hostname`); would using just those two be sufficient for automated testing? That would avoid the need for running as root, and I suspect might work on all OSs.

Testing using a VM is fine, and interfaces are a dime a dozen on those.

Ok, but requiring that does preclude automated testing on Jenkins (the APS one at least).

> would using just those two be sufficient for automated testing?

The test process as currently written wants two interfaces which would be found with EPICS_CA_AUTO_ADDR_LIST=YES. Since the loopback will now be included in this list, I think this would be sufficient.

> at would avoid the need for running as root

On reflection I was making an implicit assumption that the test code would have to change the network configuration, which isn't strictly required. Rather it could inspect, and SKIP if the host doesn't meet some requirements.

Are you working on some automated tests for this branch, or should I merge it?

I have some ideas, but haven't started writing. Go ahead and merge.

I did a partial test.

- The multicast issue seems to be fixed.
- Cygwin still fails miserably

Test results as follows

---- snip ----

Test Report rsrvbindiface on Windows 7 - ONLY PARTIAL TEST

Test Plan https://wiki-ext.aps.anl.gov/epics/index.php/RSRV_Configuration_Test_Plan
Version 2016-02-16T14:44

Tested 2016-02-26T20:00 by Ralph
                duration 1.5 hrs
Version lp:~epics-core/epics-base/rsrvbindiface revision 12765
                    plus commit 12716..12717 from lp:~epics-core/epics-base/3.15
Targets MSVC 12.0 (amd64), MSVC (x86), Cygwin (x86_64), MinGW32 (x86)
                dynamic builds
Machine Latitude E7240 (intel i3) running Windows 7

Test #1 Defaults
                            MSVC 64 MSVC 32 Cygwin MinGW32
    starting second IOC pass pass pass pass
    client get pass pass pass pass
    epicsParamShow pass pass pass pass
    casr 1 pass pass pass pass
    dbcar "" 8 pass pass pass pass

Note on Cygwin:
- warning at boot time
    Warning: Duplicate EPICS CA Address list entry "169.254.255.255:5065" discarded
- casr and dbcar (UDP search destination) show this interface address (link-local)
  in addition to the two regular interfaces

Test #2 Client address list only
                            MSVC 64 MSVC 32 Cygwin MinGW32
    starting second IOC pass pass pass pass
    client get pass pass pass pass
    epicsParamShow pass pass pass pass
    casr 1 pass pass pass pass
    dbcar "" 8 pass pass pass pass

Note on Cygwin:
- warning at boot time
    Warning: Duplicate EPICS CA Address list entry "169.254.255.255:5065" discarded
- casr shows this interface address in addition to the two regular interfaces

Test #3 Client and Server on a single interface
                            MSVC 64 MSVC 32 Cygwin MinGW32
    starting second IOC pass pass FAIL pass
    client get pass pass FAIL pass
    epicsParamShow pass pass FAIL pass
    casr 1 pass pass FAIL pass
    dbcar "" 8 pass pass FAIL pass

Failure on Cygwin:

- error at boot time
    [cas warnings in case of second IOC]
    CAS: UDP Socket bcast bind error: "Cannot assign requested address"
  IOC hangs after that message

Test #4 Client and Server on multiple interfaces
                            MSVC 64 MSVC 32 Cygwin MinGW32
    starting second IOC skip skip FAIL skip
    client get skip skip FAIL skip
    epicsParamShow skip skip FAIL skip
    casr 1 skip skip FAIL skip
    dbcar "" 8 skip skip FAIL skip

Failure on Cygwin:
-...

Re: Automation

I would not test this as part of a unit test, but move it into a separate job.
In that case it can be executed on a selected (group of) builder(s), which can easily be configured to be the VM kind with two interfaces.

> #if !(defined(_WIN32) || defined(__CYGWIN32__))

So clearly __CYGWIN32__ is not the right macro to test. Can you try some others (maybe __CYGWIN__) and/or send me the output of:

> touch empty.c
> cpp -dM empty.c

> So clearly __CYGWIN32__ is not the right macro to test.

We define our own flags for most architectures, please look for and use those instead of trying to guess what the compiler might define:

tux% cd configure/os
tux% grep OP_SYS_CPPFLAGS *
CONFIG.Common.cygwin-x86:OP_SYS_CPPFLAGS += -DCYGWIN32
CONFIG.Common.freebsdCommon:OP_SYS_CPPFLAGS += -D_BSD_SOURCE
CONFIG.Common.freebsdCommon:OP_SYS_CPPFLAGS += -Dfreebsd
CONFIG.Common.linuxCommon:OP_SYS_CPPFLAGS += -Dlinux
CONFIG.Common.RTEMS: $(USR_CPPFLAGS) $(CMD_CPPFLAGS) $(ARCH_DEP_CPPFLAGS) $(OP_SYS_CPPFLAGS)\
CONFIG.Common.solaris-sparc:OP_SYS_CPPFLAGS += -DSOLARIS=$(SOLARIS_VERSION) $(COMPILER_CPPFLAGS)
CONFIG.Common.solaris-x86:OP_SYS_CPPFLAGS += -DSOLARIS=$(SOLARIS_VERSION) $(COMPILER_CPPFLAGS)
CONFIG.Common.UnixCommon:OP_SYS_CPPFLAGS += -DUNIX
CONFIG.Common.vxWorksCommon:OP_SYS_CPPFLAGS += -DvxWorks=vxWorks
CONFIG.Common.win32-x86-mingw:OP_SYS_CPPFLAGS = -D_MINGW
CONFIG.darwinCommon.darwinCommon:OP_SYS_CPPFLAGS += -Ddarwin

I'm not guessing Andrew, I copied shareLib.h. Should I change this to 'CYGWIN32' as well?

There is also https://sourceforge.net/p/predef/wiki/OperatingSystems/ which lists '__CYGWIN__'.

This might explain some strange Cygwin DLL build issues in the past.

I am changing all the Cygwin macro tests to use __CYGWIN__, and will add a comment deprecating our CYGWIN32 macro.

Ah, if I run "cpp -dM -m32 empty.c" it defines __CYGWIN32__ as well as __CYGWIN__ so they evidently removed that when they added 64-bit support. There is no __CYGWIN64__ macro.